大数据集群安装系列2：Hadoop HA 模式安装
1,基本环境配置：依赖zookeeper，
请参看zookeeper 的安装：
大数据集群安装系列1：Zookeeper 的安装
欢迎留言哦，欢迎提问。
2,hadoop HA 模式安装
2.1，下载相应的hadoop 安装包，本文是2.7.2， 放到服务器目录下，解压，类似如下：
[root@master /opt/bigdata/component]# pwd
/opt/bigdata/component
[root@master /opt/bigdata/component]# ls
azkaban.tar.gz GsFaceLib.tar.gz hadoop.tar.gz hbase.tar.gz kafka.tar.gz spark.tgz zookeeper
elastic.tar.gz hadoop haproxy-1.7.9.tar.gz hive.tar.gz rocketmq.zip zabbix-3.4.6.tar.gz zookeeper.tar.gz
[root@master /opt/bigdata/component]#
2.2 配置cores-site.xml：请根据具体提示修改
<configuration>
<property>
<name>dfs.nameservicesname>
<value>haclustervalue>
property>
<property>
<name>dfs.ha.namenodes.haclustername>
<value>nn1,nn2value>
property>
<property>
<name>dfs.namenode.rpc-address.hacluster.nn1name>
<value>master:9000value>
property>
<property>
<name>dfs.namenode.http-address.hacluster.nn1name>
<value>master:50070value>
property>
<property>
<name>dfs.namenode.rpc-address.hacluster.nn2name>
<value>workerI:9000value>
property>
<property>
<name>dfs.namenode.http-address.hacluster.nn2name>
<value>workerI:50070value>
property>
<property>
<name>dfs.namenode.shared.edits.dirname>
<value>qjournal://workerII:8485;workerI:8485;master:8485;/haclustervalue>
property>
<property>
<name>dfs.journalnode.edits.dirname>
<value>/opt/bigdata/Hadoop/hadoop/dfs_journalnode_edits_dirvalue>
property>
<property>
<name>dfs.ha.automatic-failover.enabledname>
<value>truevalue>
property>
<property>
<name>dfs.client.failover.proxy.provider.haclustername>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvidervalue>
property>
<property>
<name>dfs.ha.fencing.methodsname>
<value>
sshfence
shell(/bin/true)
value>
property>
<property>
<name>dfs.ha.fencing.ssh.private-key-filesname>
<value>/root/.ssh/id_rsavalue>
property>
<property>
<name>dfs.ha.fencing.ssh.connect-timeoutname>
<value>30000value>
property>
<property>
<name>dfs.replicationname>
<value>3value>
property>
2.4 配置yarn-site.xml
-->
<configuration>
<property>
<name>yarn.resourcemanager.ha.enabledname>
<value>truevalue>
property>
<property>
<name>yarn.resourcemanager.cluster-idname>
<value>yrcvalue>
property>
<property>
<name>yarn.resourcemanager.ha.rm-idsname>
<value>rm1,rm2value>
property>
<property>
<name>yarn.resourcemanager.hostname.rm1name>
<value>mastervalue>
property>
<property>
<name>yarn.resourcemanager.hostname.rm2name>
<value>workerIvalue>
property>
<property>
<name>yarn.resourcemanager.zk-addressname>
<value>workerII:2181,workerI:2181,master:2181,value>
property>
<property>
<name>yarn.nodemanager.aux-servicesname>
<value>mapreduce_shuffle,spark_shufflevalue>
property>
<property>
<name>yarn.nodemanager.resource.memory-mbname>
<value>19000value>
property>
<property>
<name>yarn.scheduler.minimum-allocation-mbname>
<value>8192value>
property>
<property>
<name>yarn.scheduler.maximum-allocation-mbname>
<value>19000value>
property>
<property>
<name>yarn.app.mapreduce.am.resource.mbname>
<value>8192value>
property>
<property>
<name>yarn.app.mapreduce.am.command-optsname>
<value>-Xmx6553mvalue>
property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1name>
<value>master:8088value>
property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2name>
<value>workerI:8088value>
property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle.classname>
<value>org.apache.spark.network.yarn.YarnShuffleServicevalue>
property>
<property>
<name>spark.shuffle.service.portname>
<value>7337value>
property>
configuration>
2.6 修改yarn-env.sh 和hadoop-env.sh 中java 的路径， 以及配置slaves 节点
export JAVA_HOME=/opt/bigdata/basic/jdk
[root@master /opt/bigdata/component/hadoop/etc/hadoop]# vim slaves
[root@master /opt/bigdata/component/hadoop/etc/hadoop]# cat slaves
master
workerI
workerII
2.7 分发文件：
scp -r hadoop workerI:pwd
scp -r hadoop workerII:pwd
3，安装好后，接下来就是启动了
3.1 在master 节点，首先格式化zk（此时请确保zookeeper 已经启动）
在$HADOOP_INSTALL_HOME/bin
./hdfs zkfc -formatZK -force
3.2 启动zkfc(每个节点都需要这样做)
在$HADOOP_INSTALL_HOME/sbin
./hadoop-daemon.sh start zkfc
3.3 启动journalnode（每个节点）
在$HADOOP_INSTALL_HOME/sbin
./hadoop-daemon.sh start journalnode
3.4 格式化hdfs 文件系统
在$HADOOP_INSTALL_HOME/bin
./hadoop namenode -format -force
3.5 在master 节点，进行启动hdfs
在$HADOOP_INSTALL_HOME/sbin
./start-dfs.sh
3.6 启动yarn
在$HADOOP_INSTALL_HOME/sbin
./start-yarn.sh
3.7 在备用节点，启动备用namenode 和 备用resourcemanager
本文备用节点是workerI
在HADOOP_INSTALL_HOME/sbin
yarn-daemon.sh start resourcemanager
在$HADOOP_INSTALL_HOME/sbin
hdfs namenode -bootstrapStandby
在$HADOOP_INSTALL_HOME/sbin
hadoop-daemon.sh start namenode3.8 至此，所有hdfs 内容已经启动，具体进程如下：
[root@workerI ~]# jps
1969 WrapperSimpleApp
6961 Jps
5682 JournalNode
5956 DFSZKFailoverController
5527 QuorumPeerMain
5832 DataNode
6057 NodeManager
6203 ResourceManager
[root@workerI ~]#
[root@master ~]# jps
6560 DFSZKFailoverController
6644 JournalNode
7460 ResourceManager
7112 DataNode
7563 NodeManager
8396 Jps
2076 WrapperSimpleApp
6957 NameNode
6174 QuorumPeerMain
[root@workerII ~]# jps
4067 NodeManager
3923 DataNode
3848 JournalNode
2200 WrapperSimpleApp
4606 Jps
3679 QuorumPeerMain
(请不要在一Jps 和WrapperSimpleApp 这两个不需要关注 )