Hadoop高可用集群（HA）

阅读更多

一、集群的规划
Zookeeper集群：
192.168.157.12 （bigdata12）
192.168.157.13 （bigdata13）
192.168.157.14 （bigdata14）

Hadoop集群：
192.168.157.12 （bigdata12）   NameNode1      ResourceManager1     Journalnode
192.168.157.13 （bigdata13）   NameNode2      ResourceManager2     Journalnode
192.168.157.14 （bigdata14）   DataNode1      NodeManager1
192.168.157.15 （bigdata15）   DataNode2      NodeManager2

二、准备工作
1、安装JDK
2、配置环境变量
3、配置免密码登录
4、配置主机名

三、配置Zookeeper（在192.168.157.12安装）
在主节点（hadoop112）上配置ZooKeeper
（*）配置/root/training/zookeeper-3.4.6/conf/zoo.cfg文件
dataDir=/root/training/zookeeper-3.4.6/tmp

server.1=bigdata12:2888:3888
server.2=bigdata13:2888:3888
server.3=bigdata14:2888:3888

（*）在/root/training/zookeeper-3.4.6/tmp目录下创建一个myid的空文件
echo 1 > /root/training/zookeeper-3.4.6/tmp/myid

（*）将配置好的zookeeper拷贝到其他节点，同时修改各自的myid文件
scp -r /root/training/zookeeper-3.4.6/ bigdata13:/root/training
scp -r /root/training/zookeeper-3.4.6/ bigdata14:/root/training

四、安装Hadoop集群（在bigdata12上安装）
1、修改hadoo-env.sh
export JAVA_HOME=/root/training/jdk1.8.0_144

2、修改core-site.xml



fs.defaultFS
hdfs://ns1




hadoop.tmp.dir
/root/training/hadoop-2.7.3/tmp




ha.zookeeper.quorum
bigdata12:2181,bigdata13:2181,bigdata14:2181



3、修改hdfs-site.xml（配置这个nameservice中有几个namenode）

   

dfs.nameservices
ns1




dfs.ha.namenodes.ns1
nn1,nn2




dfs.namenode.rpc-address.ns1.nn1
bigdata12:9000



dfs.namenode.http-address.ns1.nn1
bigdata12:50070




dfs.namenode.rpc-address.ns1.nn2
bigdata13:9000



dfs.namenode.http-address.ns1.nn2
bigdata13:50070




dfs.namenode.shared.edits.dir
qjournal://bigdata12:8485;bigdata13:8485;/ns1



dfs.journalnode.edits.dir
/root/training/hadoop-2.7.3/journal




dfs.ha.automatic-failover.enabled
true




dfs.client.failover.proxy.provider.ns1
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider




dfs.ha.fencing.methods

sshfence
shell(/bin/true)





dfs.ha.fencing.ssh.private-key-files
/root/.ssh/id_rsa




dfs.ha.fencing.ssh.connect-timeout
30000



4、修改mapred-site.xml


mapreduce.framework.name
yarn



5、修改yarn-site.xml

   

   yarn.resourcemanager.ha.enabled
   true


       

   yarn.resourcemanager.cluster-id
   yrc




   yarn.resourcemanager.ha.rm-ids
   rm1,rm2




   yarn.resourcemanager.hostname.rm1
   bigdata12


   yarn.resourcemanager.hostname.rm2
   bigdata13




   yarn.resourcemanager.zk-address
   bigdata12:2181,bigdata13:2181,bigdata14:2181



   yarn.nodemanager.aux-services
   mapreduce_shuffle



6、修改slaves
bigdata14
bigdata15

7、将配置好的hadoop拷贝到其他节点
scp -r /root/training/hadoop-2.7.3/ root@bigdata13:/root/training/
scp -r /root/training/hadoop-2.7.3/ root@bigdata14:/root/training/
scp -r /root/training/hadoop-2.7.3/ root@bigdata15:/root/training/

五、启动Zookeeper集群

六、在bigdata12和bigdata13上启动journalnode
hadoop-daemon.sh start journalnode

七、格式化HDFS（在bigdata12上执行）
1. hdfs namenode -format
2. 将/root/training/hadoop-2.7.3/tmp拷贝到bigdata13的/root/training/hadoop-2.7.3/tmp下
3. 格式化zookeeper
   hdfs zkfc -formatZK
   日志：17/07/13 00:34:33 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns1 in ZK.
 
  
八、在bigdata12上启动Hadoop集群
    start-all.sh

日志：
Starting namenodes on [bigdata12 bigdata13]
bigdata12: starting namenode, logging to /root/training/hadoop-2.4.1/logs/hadoop-root-namenode-hadoop113.out
bigdata13: starting namenode, logging to /root/training/hadoop-2.4.1/logs/hadoop-root-namenode-hadoop112.out
bigdata14: starting datanode, logging to /root/training/hadoop-2.4.1/logs/hadoop-root-datanode-hadoop115.out
bigdata15: starting datanode, logging to /root/training/hadoop-2.4.1/logs/hadoop-root-datanode-hadoop114.out

bigdata13: starting zkfc, logging to /root/training/hadoop-2.7.3/logs/hadoop-root-zkfc-bigdata13.out
bigdata12: starting zkfc, logging to /root/training/hadoop-2.7.3/logs/hadoop-root-zkfc-bigdata12.out


bigdata13上的ResourceManager需要单独启动
命令：yarn-daemon.sh start resourcemanager

• 
• 大小: 75.4 KB

• 查看图片附件