十六、YARN和MapReduce配置

1、部署前提

（1）配置前提

已经配置好Hadoop集群。

配置内容：

（2）部署说明

（3）集群规划

2、修改配置文件

MapReduce

（1）修改mapred-env.sh配置文件

export JAVA_HOME=/export/server/jdk
export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1000
export HADOOP_MAPRED_ROOT_LOGGER=INFO,RFA

（2）修改mapred-site.xml配置文件

  
    mapreduce.framework.name
    yarn
    
  
  
    mapreduce.jobhistory.address
    node1:10020
    
  
  
    mapreduce.jobhistory.webapp.address
    node1:19888
    
  
  
    mapreduce.jobhistory.intermediate-done-dir
    /data/mr-history/tmp
    
  
  
    mapreduce.jobhistory.done-dir
    /data/mr-history/done
    
  
  
    yarn.app.mapreduce.am.env
    HADOOP_MAPRED_HOME=$HADOOP_HOME
  
  
    mapreduce.map.env
    HADOOP_MAPRED_HOME=$HADOOP_HOME
  
  
    mapreduce.reduce.env
    HADOOP_MAPRED_HOME=$HADOOP_HOME
  

yarn

（1）修改yarn-env.sh文件

export JAVA_HOME=/export/server/jdk
export HADOOP_HOME=/export/server/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
# export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
# export YARN_LOG_DIR=$HADOOP_HOME/logs/yarn
export HADOOP_LOG_DIR=$HADOOP_HOME/logs

（2）修改yarn-site.xml文件

    yarn.log.server.url
    http://node1:19888/jobhistory/logs
    


  
    yarn.web-proxy.address
    node1:8089
    proxy server hostname and port
  


  
    yarn.log-aggregation-enable
    true
    Configuration to enable or disable log aggregation
  

  
    yarn.nodemanager.remote-app-log-dir
    /tmp/logs
    Configuration to enable or disable log aggregation
  



  
    yarn.resourcemanager.hostname
    node1
    
  

  
    yarn.resourcemanager.scheduler.class
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
    
  

  
    yarn.nodemanager.local-dirs
    /data/nm-local
    Comma-separated list of paths on the local filesystem where intermediate data is written.
  


  
    yarn.nodemanager.log-dirs
    /data/nm-log
    Comma-separated list of paths on the local filesystem where logs are written.
  


  
    yarn.nodemanager.log.retain-seconds
    10800
    Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled.
  



  
    yarn.nodemanager.aux-services
    mapreduce_shuffle
    Shuffle service that needs to be set for Map Reduce applications.
  

 分发配置文件

 3、开启YARN服务器集群

（1）node1节点，以Hadoop用户，执行如下语句：

//开启Hadoop集群
start-dfs.sh
//开启yarn集群
start-yarn.sh
//因为代理服务器已经在配置文件中配置完毕，但历史服务器需要手动开启
//开启历史服务器
mapred --daemon start historyserver

（2）执行结果展示、

（3）查看YARN的Web-UI页面

        打开本地浏览器，输入node1:8088，即可打开本地YARN的Web-UI页面。

4、总结