Hadoop-2.2.0中文文档——Hadoop MapReduce 下一代 —配置一个单节点集群

Mapreduce 包

你需从发布页面获得MapReduce tar包。若不能,你要将源码打成tar包。
 
   
$ mvn clean install -DskipTests
$ cd hadoop-mapreduce-project
$ mvn clean install assembly:assembly -Pnative
 
   

注意:你需要安装有protoc 2.5.0。

忽略本地建立mapreduce,你可以在maven中省略-Pnative参数。tar包应该在target/directory。

配置环境

假设你已经安装hadoop-common/hadoop-hdfs,并且输出了$HADOOP_COMMON_HOME/$HADOOP_HDFS_HOME,解压hadoop mapreduce 包,配置环境变量$HADOOP_MAPRED_HOME到要安装的目录。$HADOOP_YARN_HOME的配置和$HADOOP_MAPRED_HOME一样.

注意:下面的操作假设你已经运行了hdfs。

设置配置信息

要启动ResourceManager and NodeManager, 你必须升级配置。假设你的 $HADOOP_CONF_DIR是配置目录,并且已经安装了HDFS和core-site.xml。还有2个配置文件你必须设置mapred-site.xmlyarn-site.xml.

设置mapred-site.xml

添加下面的配置到你的mapred-site.xml.


    mapreduce.cluster.temp.dir
    
    No description
    true
  

  
    mapreduce.cluster.local.dir
    
    No description
    true
  

设置yarn-site.xml

添加下面的配置到你的yarn-site.xml.


    yarn.resourcemanager.resource-tracker.address
    host:port
    host is the hostname of the resource manager and 
    port is the port on which the NodeManagers contact the Resource Manager.
    
  

  
    yarn.resourcemanager.scheduler.address
    host:port
    host is the hostname of the resourcemanager and port is the port
    on which the Applications in the cluster talk to the Resource Manager.
    
  

  
    yarn.resourcemanager.scheduler.class
    org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
    In case you do not want to use the default scheduler
  

  
    yarn.resourcemanager.address
    host:port
    the host is the hostname of the ResourceManager and the port is the port on
    which the clients can talk to the Resource Manager. 
  

  
    yarn.nodemanager.local-dirs
    
    the local directories used by the nodemanager
  

  
    yarn.nodemanager.address
    0.0.0.0:port
    the nodemanagers bind to this port
    

  
    yarn.nodemanager.resource.memory-mb
    10240
    the amount of memory on the NodeManager in GB
  
 
  
    yarn.nodemanager.remote-app-log-dir
    /app-logs
    directory on hdfs where the application logs are moved to 
  

   
    yarn.nodemanager.log-dirs
    
    the directories used by Nodemanagers as log directories
  

  
    yarn.nodemanager.aux-services
    mapreduce_shuffle
    shuffle service that needs to be set for Map Reduce to run 
  


设置capacity-scheduler.xml

确保你放置根队列到capacity-scheduler.xml.

 
    yarn.scheduler.capacity.root.queues
    unfunded,default
  
  
  
    yarn.scheduler.capacity.root.capacity
    100
  
  
  
    yarn.scheduler.capacity.root.unfunded.capacity
    50
  
  
  
    yarn.scheduler.capacity.root.default.capacity
    50
  

运行守护进程

假设环境变量$HADOOP_COMMON_HOME,$HADOOP_HDFS_HOME,$HADOO_MAPRED_HOME,$HADOOP_YARN_HOME,$JAVA_HOME$HADOOP_CONF_DIR已经设置正确。$$YARN_CONF_DIR 的设置同 $HADOOP_CONF_DIR。

运行ResourceManager 和 NodeManager 如下:

$ cd $HADOOP_MAPRED_HOME
$ sbin/yarn-daemon.sh start resourcemanager
$ sbin/yarn-daemon.sh start nodemanager

你应该启动和运行。你可以运行randomwriter如下:

$ $HADOOP_COMMON_HOME/bin/hadoop jar hadoop-examples.jar randomwriter out

祝你好运。

你可能感兴趣的:(Hadoop-2.2.0中文文档——Hadoop MapReduce 下一代 —配置一个单节点集群)