Hadoop集群安装 (5) 配置job queue_conf/mapred-queues.xml

0.总体

(0)job queue的设置和scheduler密切相关

1) scheduler的配置见 conf/mapred-site.xml中的mapreduce.jobtracker.taskscheduler 

2) Fairshare不支持queue

Fairshare scheduler, implement their own mechanisms for collections of jobs and do not rely on queues provided by the framework

3)管理员应该认真选择 sheduler，然后选择配置对应的 queue

(1) Job队列的作用

1)组织job

2)对作业队列的策略：调度策略，管理策略

(2)对作业队列的基本操作，例如：

job submission to a specific queue, 

access control for queues, 

queue states, 

viewing configured queues and their properties 

refresh of queue properties.

(3)作业队列的类型：

Single queue

The default configuration in Map/Reduce comprises of a single queue, as supported by the default scheduler. All jobs are submitted to this default queue which maintains jobs in a priority basedFIFO order. 

Multiple single level queues

Multiple queues are defined, and jobs can be submitted to any of these queues. Different policies can be applied to these queues by schedulers that support this configuration to provide a better level of support. For example, the capacity scheduler provides ways of configuring different capacity and fairness guarantees on these queues. 

Hierarchical queues: 

Hierarchical queues are a configuration in which queues can contain other queues within them recursively. The queues that contain other queues are referred to as container queues. Queues that do not contain other queues are referred as leaf or job queues. Jobs can only be submitted to leaf queues. Hierarchical queues can potentially offer a higher level of control to administrators, as schedulers can now build a hierarchy of policies where policies applicable to a container queue can provide context for policies applicable to queues it contains. It also opens up possibilities for delegating queue administration where administration of queues in a container queue can be turned over to a different set of administrators, within the context provided by the container queue. For example, the capacity scheduler uses hierarchical queues to partition capacity of a cluster among container queues, and allowing queues they contain to divide that capacity in more ways.

(4)文件

hadoop-0.21.0/mapred/src/java/mapred-queues-default.xml,

正式配置文件在 conf/mapred-queues.xml

1. 层级作业队列(举例)

<queues aclsEnabled="$aclsEnabled">
	<queue>
		<name>$queue-name</name>
		<state>$state</state>
		
		<queue>
			<name>$child-queue1</name>
			<properties>
				<property key="$key" value="$value"/>
			</properties>
			<queue>
				<name>$grand-child-queue1</name>
				...
			</queue>
		</queue>
              
		<queue>
			<name>$child-queue2</name>
			...
		</queue>
			
		<queue>
			<name>$leaf-queue</name>
			<acl-submit-job>$acls</acl-submit-job>
			<acl-administer-jobs>$acls</acl-administer-jobs>
			<properties>
				<property key="$key" value="$value"/>
				...
			</properties>
		</queue>
	</queue>
</queues>

2.  定义完作业队列之后，可以在conf/mapred-site.xml中设置

(1)声明作业提交给哪个队列

<property>
	<name>mapreduce.job.queuename</name>
	<value>Queue-A:Queue-B:Queue-C</value>
</property>

(2) 指明调度器和算法

<property>
  <name>mapreduce.jobtracker.taskscheduler</name>
  <value>org.apache.hadoop.mapred.JobQueueTaskScheduler</value>
  <description>The class responsible for scheduling the tasks.</description>
</property>

3. 查看作业队列

(1) 通过命令行查看

hadoop queue [-list] | [-info <job-queue-name> [-showJobs]] | [-showacls]

(2)通过网页查看 

queueinfo.jsp