Hadoop入门学习笔记-第三天（Yarn高可用集群配置及计算案例）

什么是mapreduce

首先让我们来重温一下 hadoop 的四大组件：
HDFS：分布式存储系统
MapReduce：分布式计算系统
YARN： hadoop 的资源调度系统
Common： 以上三大组件的底层支撑组件，主要提供基础工具包和 RPC 框架等

Mapreduce 是一个分布式运算程序的编程框架，是用户开发“基于 hadoop 的数据分析 应用”的核心框架
Mapreduce 核心功能是将用户编写的业务逻辑代码和自带默认组件整合成一个完整的 分布式运算程序，并发运行在一个 hadoop 集群上

一、yarn单机版配置
1.修改mapred-site.xml 文件指定调度方式为yarn

mapreduce.framework.name
yarn

2.修改yarn-site.xml 配置yarn服务器

yarn.resourcemanager.hostname
nameNode10.com

  
    yarn.nodemanager.aux-services
    mapreduce_shuffle
  

3.在指定机器上面启动./start-yarn.sh

二、yarn ha高可用 集群配置
1.修改yarn-site.xml 配置yarn集群


yarn.resourcemanager.connect.retry-interval.ms
2000



yarn.resourcemanager.ha.enabled
true



yarn.resourcemanager.ha.automatic-failover.enabled
true

    yarn.resourcemanager.ha.automatic-failover.embedded
    true

    

    yarn.resourcemanager.cluster-id
    yarn-rm-cluster



    yarn.resourcemanager.ha.rm-ids
    rm1,rm2



    yarn.resourcemanager.hostname.rm1
    nameNode10.com



    yarn.resourcemanager.hostname.rm2
    dateNode30.com

    yarn.resourcemanager.recovery.enabled
    true

    yarn.resourcemanager.zk.state-store.address
    nameNode10.com:2181,dataNode20.com:2181,nameNode40.com:2181



    yarn.resourcemanager.zk-address
    nameNode10.com:2181,dataNode20.com:2181,nameNode40.com:2181

    yarn.resourcemanager.address.rm1
    nameNode10.com:8032



    yarn.resourcemanager.scheduler.address.rm1
    nameNode10.com:8034



    yarn.resourcemanager.webapp.address.rm1
    nameNode10.com:8088

yarn.resourcemanager.address.rm2 nodeNode30.com:8032 yarn.resourcemanager.scheduler.address.rm2 nodeNode30.com:8034 yarn.resourcemanager.webapp.address.rm2 nodeNode30.com:8088

    yarn.nodemanager.aux-services
    mapreduce_shuffle


    yarn.nodemanager.aux-services.mapreduce_shuffle.class
    org.apache.hadoop.mapred.ShuffleHandler

2.输入hadoop classpath 会出现以下路径(这步不配置，执行自带案例demo时会报错)
/software/hadoop-3.2.0/etc/hadoop:/software/hadoop-3.2.0/share/hadoop/common/lib/:/software/hadoop-3.2.0/share/hadoop/common/:/software/hadoop-3.2.0/share/hadoop/hdfs:/software/hadoop-3.2.0/share/hadoop/hdfs/lib/:/software/hadoop-3.2.0/share/hadoop/hdfs/:/software/hadoop-3.2.0/share/hadoop/mapreduce/lib/:/software/hadoop-3.2.0/share/hadoop/mapreduce/:/software/hadoop-3.2.0/share/hadoop/yarn:/software/hadoop-3.2.0/share/hadoop/yarn/lib/:/software/hadoop-3.2.0/share/hadoop/yarn/
将上面路径拷贝到 yarn-site.xml文件中

yarn.application.classpath
/software/hadoop-3.2.0/etc/hadoop:/software/hadoop-3.2.0/share/hadoop/common/lib/:/software/hadoop-3.2.0/share/hadoop/common/:/software/hadoop-3.2.0/share/hadoop/hdfs:/software/hadoop-3.2.0/share/hadoop/hdfs/lib/:/software/hadoop-3.2.0/share/hadoop/hdfs/:/software/hadoop-3.2.0/share/hadoop/mapreduce/lib/:/software/hadoop-3.2.0/share/hadoop/mapreduce/:/software/hadoop-3.2.0/share/hadoop/yarn:/software/hadoop-3.2.0/share/hadoop/yarn/lib/:/software/hadoop-3.2.0/share/hadoop/yarn/

3.单步骤启动
yarn-daemon.sh start resourcemanager
yarn-daemon.sh start nodemanager
4.集成启动 yarn-start.sh

5.访问控制台查看(访问备用节点会自动跳转到主节点控制台)
http://datanode30.com:8088/cluster/
http://namenode10.com:8088/cluster/
主备状态查看
yarn rmadmin -getServiceState rm1

6.执行例子
1.计算圆周率 /software/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar pi 10 10
2.统计文中每个单词个数
hadoop fs -mkdir /input
hadoop fs -put -f ./file/file1.txt /input
hadoop jar /software/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /input /output
hadoop fs -ls /output
hadoop fs -cat /output/part-r-00000

/input/hadoop-demo-wordcount.txt