【Yarn】配置公平调度器

目录

添加如下属性到yarn-site.xml

添加配置文件fair-scheduler.xml

将新的配置同步到集群所有节点,重启Yarn

创建相应的用户进行测试

提交任务

查看任务

---

• 添加如下属性到yarn-site.xml

       yarn.resourcemanager.scheduler.class
       org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

 


       yarn.scheduler.fair.allocation.file
       /export/servers/hadoop-2.6.0-cdh5.14.0/etc/hadoop/fair-scheduler.xml

 


       yarn.scheduler.fair.preemption
       true

 

       yarn.scheduler.fair.preemption.cluster-utilization-threshold
       0.8f

 


       yarn.scheduler.fair.user-as-default-queue
       true

 


       yarn.scheduler.fair.allow-undeclared-pools
       false

 

• 添加配置文件fair-scheduler.xml

30


         512mb,4vcores
         102400mb,100vcores
         100
         1.0
         fair
          
          
 
         
                   512mb,4vcores
                   30720mb,30vcores
                   100
                   fair
                   1.0
                   
                   *
         
 

         
                   512mb,4vcores
                   20480mb,20vcores
                   100
                   fair
                   2.0
                   hadoop hadoop
                   hadoop hadoop
         
 
         
                   512mb,4vcores
                   20480mb,20vcores
                   100
                   fair
                   1
                   develop develop
                   develop develop
         
 
         
                   512mb,4vcores
                   20480mb,20vcores
                   100
                   fair
                   1.5
                   test1,hadoop,develop test1
                   test1 group_businessC,supergroup
         

 

• 将新的配置同步到集群所有节点,重启Yarn

同步：

scp yarn-site.xml  fair-scheduler.xml node02:$PWD
scp yarn-site.xml  fair-scheduler.xml node03:$PWD

 

重启Yarn：

stop-yarn.sh
start-yarn.sh

 

• 创建相应的用户进行测试

useradd hadoop
passwd hadoop
groupadd supergroup
usermod -a -G supergroup hadoop    修改用户所属的附加群主
su - root -s /bin/bash -c "hdfs dfsadmin -refreshUserToGroupsMappings"
刷新用户组信息
su hadoop  切换用户

 

• 提交任务

hadoop jar /export/servers/hadoop-2.6.0-cdh5.14.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.14.0.jar pi 10 20

 

• 查看任务

http://192.168.100.201:8088/cluster/scheduler