YARN 高可用(HA)集群搭建(CDH)

详尽的视频讲解,请查看该地址:https://edu.csdn.net/course/detail/9572

HDFS 高可用集群安装完成之后,紧接着快速搞定YARN的集群搭建。

1.修改YARN配置文件

1.1修改yarn-site.xml



	yarn.resourcemanager.connect.retry-interval.ms
	2000


	yarn.resourcemanager.ha.enabled
	true



	yarn.resourcemanager.ha.automatic-failover.enabled
	true



	yarn.resourcemanager.ha.automatic-failover.embedded
	true



	yarn.resourcemanager.cluster-id
	yarn-rm-cluster



	yarn.resourcemanager.ha.rm-ids
	rm1,rm2



	yarn.resourcemanager.hostname.rm1
	hadoop01


	yarn.resourcemanager.hostname.rm2
	hadoop02


	yarn.resourcemanager.recovery.enabled
	true



	yarn.resourcemanager.zk.state-store.address
	hadoop01:2181,hadoop02:2181,hadoop03:2181



	yarn.resourcemanager.zk-address
	hadoop01:2181,hadoop02:2181,hadoop03:2181



	yarn.resourcemanager.address.rm1
	hadoop01:8032



	yarn.resourcemanager.scheduler.address.rm1
	hadoop01:8034



	yarn.resourcemanager.webapp.address.rm1
	hadoop01:8088



	yarn.resourcemanager.address.rm2
	hadoop02:8032


	yarn.resourcemanager.scheduler.address.rm2
	hadoop02:8034


	yarn.resourcemanager.webapp.address.rm2
	hadoop02:8088


	yarn.nodemanager.aux-services
	mapreduce_shuffle


	yarn.nodemanager.aux-services.mapreduce_shuffle.class
	org.apache.hadoop.mapred.ShuffleHandler



1.2修改mapred-site.xml



	mapreduce.framework.name
	yarn



2.修改过的配置文件同步到其他节点

通过如下命令,将yarn相关配置同步到slave节点。

deploy.sh mapred-site.xml /home/hadoop/app/hadoop-2.6.0-hadoop5.10.0/etc/hadoop slave

deploy.sh yarn-site.xml /home/hadoop/app/hadoop-2.6.0-hadoop5.10.0/etc/hadoop slave

3.一键启动YARN集群

主节点hadoop01通过如下命令一键启动YARN 集群。

sbin/start-yarn.sh

4.启动备用节点RM

在备用节点hadoop02单独启动ResourceManager进程。

sbin/yarn-daemon.sh start resourcemanager

5.查看RM状态

通过如下命令分别查看rm1 rm2的状态。

bin/yarn rmadmin -getServiceState rm1

bin/yarn rmadmin -getServiceState rm2

6.web界面查看yarn

通过如下地址访问web界面,了解YARN相关作业运行信息。

http://hadoop01:8088

或者

http://hadoop02:8088

7.测试运行YARN

在YARN上面运行Hadoop自带Wordcount程序

bin/hadoop jar share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.10.0.jar wordcount /test/wd.txt /test/output

通过命令行查看执行结果

bin/hdfs dfs -cat /test/output/*

如果Wordcount程序能在YARN上成功运行,说明整个Hadoop集群安装成功。

详尽的视频讲解,请查看该地址:https://edu.csdn.net/course/detail/9572

你可能感兴趣的:(大数据)