本节主要内容：
YARN环境搭建
1.系统环境：
OS：CentOS Linux release 7.5.1804 (Core)
CPU：2核心
Memory：1GB
运行用户：root
JDK版本：1.8.0_252
Hadoop版本：cdh5.16.2
2.集群各节点角色规划为：
172.26.37.245 node1.hadoop.com---->namenode，zookeeper，journalnode，hadoop-hdfs-zkfc，resourcenode，historyserver
172.26.37.246 node2.hadoop.com---->datanode，zookeeper，journalnode，nodemanager，hadoop-client，mapreduce
172.26.37.247 node3.hadoop.com---->datanode，nodemanager，hadoop-client，mapreduce
172.26.37.248 node4.hadoop.com---->namenode，zookeeper，journalnode，hadoop-hdfs-zkfc
3.环境说明：
本次追加部署
172.26.37.245 node1.hadoop.com---->resourcenode，historyserver
172.26.37.246 node2.hadoop.com---->nodemanager，hadoop-client，mapreduce
172.26.37.246 node2.hadoop.com---->nodemanager，hadoop-client，mapreduce
node1节点：负责任务调度，资源管理，和任务记录，部署resourcenode，historyserver
node2、node3节点：负责运行任务，部署nodemanager，hadoop-client，mapreduce
一.安装
node1节点
# yum -y install hadoop-yarn-resourcemanager hadoop-mapreduce-historyserver
node2、node3节点
# yum -y install hadoop-yarn-nodemanager hadoop-mapreduce hadoop-client
二.配置
1./etc/hadoop/conf/core-site.xml变更(node1、node2、node3节点)
# cp -p /etc/hadoop/conf/core-site.xml /etc/hadoop/conf/core-site.xml.20200623
# vi /etc/hadoop/conf/core-site.xml
添加如下
hadoop.proxyuser.mapred.groups
*
hadoop.proxyuser.mapred.hosts
*
2.配置YARN集群的属性及配置JobHistory服务器(node1节点)
# cp -p /etc/hadoop/conf/mapred-site.xml /etc/hadoop/conf/mapred-site.xml.20200623
# vi /etc/hadoop/conf/mapred-site.xml
mapreduce.framework.name
yarn
mapreduce.jobhistory.address
node1.hadoop.com:10020
mapreduce.jobhistory.webapp.address
node1.hadoop.com:19888
yarn.app.mapreduce.am.staging-dir
/user
3.配置YARN守护进程(node1、node2、node3节点)
配置以下服务：ResourceManager（在专用主机上）和NodeManager（在计划运行MapReduce v2作业的每台主机上）。
# cp /etc/hadoop/conf/yarn-site.xml /etc/hadoop/conf/yarn-site.xml.20200623
# vi /etc/hadoop/conf/yarn-site.xml
变更后为以下内容
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce_shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
yarn.log-aggregation-enable
true
List of directories to store localized files in.
yarn.nodemanager.local-dirs
file:///var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir
Where to store container logs.
yarn.nodemanager.log-dirs
file:///var/log/hadoop-yarn/containers
Where to aggregate logs to.
yarn.nodemanager.remote-app-log-dir
hdfs://cluster1/var/log/hadoop-yarn/apps
Classpath for typical applications.
yarn.application.classpath
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
yarn.resourcemanager.hostname
node1.hadoop.com
4.建立yarn在hdfs上的日志文件夹
# sudo -u hdfs hadoop fs -mkdir -p /var/log/hadoop-yarn
# sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
5.创建实验环境
创建/user目录和及其history子目录；设置权限
# sudo -u hdfs hadoop fs -mkdir /user
# sudo -u hdfs hadoop fs -mkdir -p /user/cloudera
# sudo -u hdfs hadoop fs -chmod -R 1777 /user/cloudera
# sudo -u hdfs hadoop fs -chown mapred:hadoop /user/cloudera
# sudo -u hdfs hadoop fs -mkdir -p /user/hdfs
# sudo -u hdfs hadoop fs -chmod -R 1777 /user/hdfs
# sudo -u hdfs hadoop fs -chown mapred:hadoop /user/hdfs
# sudo -u hdfs hadoop fs -mkdir -p /user/history
# sudo -u hdfs hadoop fs -chmod -R 1777 /user/history
# sudo -u hdfs hadoop fs -chown mapred:hadoop /user/history
# sudo -u hdfs hdfs dfs -ls /
Found 3 items
drwxrwxrwt - hdfs supergroup 0 2020-06-25 11:47 /tmp
drwxr-xr-x - hdfs supergroup 0 2020-06-25 11:57 /user
drwxr-xr-x - hdfs supergroup 0 2020-06-25 11:49 /var
# sudo -u hdfs hadoop dfs -ls /user
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Found 3 items
drwxrwxrwt - mapred hadoop 0 2020-06-25 11:57 /user/cloudera
drwxrwxrwt - mapred hadoop 0 2020-06-25 11:56 /user/hdfs
drwxrwxrwt - mapred hadoop 0 2020-06-25 11:59 /user/history
6.启动YARN和MapReduce JobHistory服务器
要启动YARN，必须先启动ResourceManager和NodeManager服务：
同时确保在启动NodeManager服务之前先启动ResourceManager。
在ResourceManager系统上(node1节点)：
# service hadoop-yarn-resourcemanager start
# service hadoop-yarn-resourcemanager status
在每个NodeManager系统上（node2/3节点）：
# service hadoop-yarn-nodemanager start
# service hadoop-yarn-nodemanager status
启动MapReduce JobHistory服务器(node1节点)
# service hadoop-mapreduce-historyserver start
# service hadoop-mapreduce-historyserver status
7.验证history服务
http://172.26.37.245:19888/jobhistory