搭建Hadoop-HA + ZooKeeper + Yarn环境

前提：搭建Hadoop-HA + ZooKeeper环境

node01	node02	node03	node04
NameNode01	NameNode02	NameNode03
	DataNode01	DataNode02	DataNode03
JournalNode01	JournalNode02	JournalNode03
	ZooKeeper01	ZooKeeper02	ZooKeeper03
ZooKeeperFailoverController01	ZooKeeperFailoverController02	ZooKeeperFailoverController03
		ResourceManager01	ResourceManager02
	NodeManager01	NodeManager02	NodeManager03

1. 配置node01、node02、node03、node04上的Yarn

在node01上修改/opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml：
vim /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml
添加：

  
    yarn.nodemanager.aux-services
    mapreduce_shuffle
  
  
    yarn.nodemanager.env-whitelist
    JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME 
  
  
    yarn.resourcemanager.ha.enabled
    true
  
  
    yarn.resourcemanager.cluster-id
    yarnResourceManagerHACluster
  
  
    yarn.resourcemanager.ha.rm-ids
    RM01,RM02
  
  
    yarn.resourcemanager.hostname.RM01
    node03
  
  
    yarn.resourcemanager.hostname.RM02
    node04
  
  
    yarn.resourcemanager.webapp.address.RM01
    node03:8088
  
  
    yarn.resourcemanager.webapp.address.RM02
    node04:8088
  
  
    yarn.resourcemanager.zk-address
    node02:2181,node03:2181,node04:2181
  

在node01上修改/opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml：
vim /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml
添加：

  
    mapreduce.framework.name
    yarn
  
  
    mapreduce.application.classpath
    $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
  

将node01上的/opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml、/opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml拷贝到node02、node03、node04：
scp /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml node02:/opt/hadoop/hadoop-3.1.1/etc/hadoop/ && scp /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml node03:/opt/hadoop/hadoop-3.1.1/etc/hadoop/ && scp /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml node04:/opt/hadoop/hadoop-3.1.1/etc/hadoop/

2. 配置node01、node02、node03、node04上的环境变量

在node01上修改/etc/profile：
vim /etc/profile
添加：

export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop
export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1

在node02上修改/etc/profile
vim /etc/profile
添加：

export YARN_NODEMANAGER_USER=root
export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop
export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1

在node03上修改/etc/profile
vim /etc/profile
添加：

export YARN_NODEMANAGER_USER=root
export YARN_RESOURCEMANAGER_USER=root
export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop
export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1

在node04上修改/etc/profile
vim /etc/profile
添加：

export YARN_NODEMANAGER_USER=root
export YARN_RESOURCEMANAGER_USER=root
export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop
export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1

在node01、node02、node03、node04上运行：
. /etc/profile

3. 启动ZooKeeper

在node02、node03、node04上运行：
zkServer.sh start

4. 启动ResourceManager和NodeManager

在node3/node4上运行：
start-yarn.sh

5. 启动Hadoop

在node01/node02/node03/node04上运行：
start-dfs.sh

6. 查看进程

在node01、node02、node03、node04上运行：
jps

7. 访问网页

NameNode01：http://192.168.163.191:9870
NameNode02：http://192.168.163.192:9870
NameNode03：http://192.168.163.193:9870
DataNode01：http://192.168.163.192:9864
DataNode02：http://192.168.163.193:9864
DataNode03：http://192.168.163.194:9864
ResourceManager01：http://192.168.163.193:8088
ResourceManager02：http://192.168.163.194:8088