搭建CDH 阿里云 (Step 2: 启动yarn)

Yarn HA 文档

Yarn HA 文档

配置 Yarn 分配集群资源

文档地址

yarn-site.xml 配置

  

   Classpath for typical applications.
   yarn.application.classpath
   
      $HADOOP_CONF_DIR,
      $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
      $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
      $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
      $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
   


  yarn.log-aggregation-enable
  true


  yarn.nodemanager.local-dirs
  file:///data/1/yarn/local,file:///data/2/yarn/local,file:///data/3/yarn/local


  yarn.nodemanager.log-dirs
  file:///data/1/yarn/logs,file:///data/2/yarn/logs,file:///data/3/yarn/logs


  yarn.nodemanager.remote-app-log-dir
  /var/log/hadoop-yarn/apps




  
    yarn.resourcemanager.connect.retry-interval.ms
    2000
  
  
    yarn.resourcemanager.ha.enabled
    true
  
  
    yarn.resourcemanager.ha.automatic-failover.enabled
    true
  
  
    yarn.resourcemanager.ha.automatic-failover.embedded
    true
  
  
    yarn.resourcemanager.cluster-id
    xiwu-yarn-rm-cluster
  
  
    yarn.resourcemanager.ha.rm-ids
    rm1,rm2
  
  
    yarn.resourcemanager.ha.id
    rm1
  
  
    yarn.resourcemanager.scheduler.class
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
  
  
    yarn.scheduler.fair.allow-undeclared-pools
    true
  
  
    yarn.scheduler.fair.user-as-default-queue
    true
  
  
    yarn.scheduler.fair.preemption
    true
  
  
    yarn.scheduler.fair.preemption.cluster-utilization-threshold
    0.8
  

  
    yarn.resourcemanager.recovery.enabled
    true
  
  
    yarn.resourcemanager.store.class
    org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
  
  
    yarn.resourcemanager.zk-address
    worker1:2181,worker2:2181,worker3:2181
  
  
    yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms
    5000
  

  
    yarn.resourcemanager.address.rm1
    worker1:23140
  
  
    yarn.resourcemanager.scheduler.address.rm1
    worker1:23130
  
  
    yarn.resourcemanager.webapp.https.address.rm1
    worker1:23189
  
  
    yarn.resourcemanager.webapp.address.rm1
    worker1:23188
  
  
    yarn.resourcemanager.resource-tracker.address.rm1
    worker1:23125
  
  
    yarn.resourcemanager.admin.address.rm1
    worker1:23141
  

  
    yarn.resourcemanager.address.rm2
    worker2:23140
  
  
    yarn.resourcemanager.scheduler.address.rm2
    worker2:23130
  
  
    yarn.resourcemanager.webapp.https.address.rm2
    worker2:23189
  
  
    yarn.resourcemanager.webapp.address.rm2
    worker2:23188
  
  
    yarn.resourcemanager.resource-tracker.address.rm2
    worker2:23125
  
  
    yarn.resourcemanager.admin.address.rm2
    worker2:23141
  
  
    Address where the localizer IPC is.
    yarn.nodemanager.localizer.address
    0.0.0.0:23344
  
  
    NM Webapp address.
    yarn.nodemanager.webapp.address
    0.0.0.0:23999
  
  
    yarn.nodemanager.aux-services
    mapreduce_shuffle
  
  
    yarn.nodemanager.aux-services.mapreduce.shuffle.class
    org.apache.hadoop.mapred.ShuffleHandler
  
  
    yarn.nodemanager.local-dirs
    /tmp/pseudo-dist/yarn/local
  
  
    yarn.nodemanager.log-dirs
    /tmp/pseudo-dist/yarn/log
  
  
    mapreduce.shuffle.port
    23080
  

PS. yarn.resourcemanager.ha.id 请注意
在两台master机器上 配置里的value要不一样 rm1 or rm2

启动 NodeManager 服务的机器上

sudo mkdir -p /data/1/yarn/local /data/2/yarn/local /data/3/yarn/local /data/4/yarn/local
sudo mkdir -p /data/1/yarn/logs /data/2/yarn/logs /data/3/yarn/logs /data/4/yarn/logs
sudo chown -R yarn:yarn /data/1/yarn/local /data/2/yarn/local /data/3/yarn/local /data/4/yarn/local
sudo chown -R yarn:yarn /data/1/yarn/logs /data/2/yarn/logs /data/3/yarn/logs /data/4/yarn/logs

mapred-site.xml

  mapreduce.framework.name
  yarn



  mapreduce.jobhistory.address
  worker3:10020



  mapreduce.jobhistory.webapp.address
  worker3:19888



  
  hadoop.proxyuser.mapred.groups
  *



  
  hadoop.proxyuser.mapred.hosts
  *



  
  yarn.app.mapreduce.am.staging-dir
  /user



  mapreduce.jobhistory.done-dir
  /user/history/done



  mapreduce.jobhistory.intermediate-done-dir
  /user/history/intermediate

添加文件夹和配置权限

sudo -u hdfs hadoop fs -mkdir -p /user/history
sudo -u hdfs hadoop fs -chmod -R 1777 /user/history
sudo -u hdfs hadoop fs -mkdir -p /user/history/done
sudo -u hdfs hadoop fs -chmod -R 750 /user/history/done
sudo -u hdfs hadoop fs -mkdir -p /user/history/intermediate
sudo -u hdfs hadoop fs -chmod -R 1777 /user/history/intermediate
sudo -u hdfs hadoop fs -chown mapred:hadoop /user/history

sudo -u hdfs hadoop fs -mkdir /user/root
sudo -u hdfs hadoop fs -chown root /user/root