hadoop mapred-queue-acls 配置

    hadoop作业提交时可以指定相应的队列，例如:-Dmapred.job.queue.name=queue2
通过对mapred-queue-acls.xml和mapred-site.xml配置可以对不同的队列实现不同用户的提交权限.
先编辑mapred-site.xml，修改配置如下(增加四个队列):

<property>
  <name>mapred.queue.namesname>
  <value>default,queue1,queue2,queue3,queue4value>
  <description> Comma separated list of queues configured for this jobtracker.
    Jobs are added to queues and schedulers can configure different
    scheduling properties for the various queues. To configure a property
    for a queue, the name of the queue must match the name specified in this
    value. Queue properties that are common to all schedulers are configured
    here with the naming convention, mapred.queue.$QUEUE-NAME.$PROPERTY-NAME,
    for e.g. mapred.queue.default.submit-job-acl.
    The number of queues configured in this parameter could depend on the
    type of scheduler being used, as specified in
    mapred.jobtracker.taskScheduler. For example, the JobQueueTaskScheduler
    supports only a single queue, which is the default configured here.
    Before adding more queues, ensure that the scheduler you've configured
    supports multiple queues.
  description>
property>

修改生效后通过jobtrack界面可以看到配置的队列信息:

要对队列进行控制， 还需要编辑mapred-queue-acls.xml文件

<property>
  <name>mapred.queue.queue1.acl-submit-jobname>
  <value>' 'value>
  <description> Comma separated list of user and group names that are allowed
   to submit jobs to the 'default' queue. The user list and the group list
   are separated by a blank. For e.g. user1,user2 group1,group2.
   If set to the special value '*', it means all users are allowed to
   submit jobs. If set to ' '(i.e. space), no user will be allowed to submit
   jobs.

   It is only used if authorization is enabled in Map/Reduce by setting the
   configuration property mapred.acls.enabled to true.
   Irrespective of this ACL configuration, the user who started the cluster and
   cluster administrators configured via
   mapreduce.cluster.administrators can submit jobs.
  description>
property>

  要配置多个队列， 只需要重复添加上面配置信息，修改队列名称和value值，为方便测试，queue1禁止所有用户向其提交作业. 
   要使该配置生效， 还需要修改mapred-site.xml,将mapred.acls.enabled值设置为true

<property>
  <name>mapred.acls.enabledname>
  <value>truevalue>
  <description> Specifies whether ACLs should be checked
    for authorization of users for doing various queue and job level operations.
    ACLs are disabled by default. If enabled, access control checks are made by
    JobTracker and TaskTracker when requests are made by users for queue
    operations like submit job to a queue and kill a job in the queue and job
    operations like viewing the job-details (See mapreduce.job.acl-view-job)
    or for modifying the job (See mapreduce.job.acl-modify-job) using
    Map/Reduce APIs, RPCs or via the console and web user interfaces.
  description>
property>

 重启hadoop, 使配置生效, 接下来拿hive进行测试:
先使用queue2队列:

set mapred.job.queue.name=queue2;
hive>
    > select count(*) from t_aa_pc_log;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201205211843_0002, Tracking URL = http://192.168.189.128:50030/jobdetails.jsp?jobid=job_201205211843_0002
Kill Command = /opt/app/hadoop-0.20.2-cdh3u3/bin/hadoop job  -Dmapred.job.tracker=192.168.189.128:9020 -kill job_201205211843_0002
2012-05-21 18:45:01,593 Stage-1 map = 0%,  reduce = 0%
2012-05-21 18:45:04,613 Stage-1 map = 100%,  reduce = 0%
2012-05-21 18:45:12,695 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201205211843_0002
OK
136003
Time taken: 14.674 seconds
hive>

 作业成功完成

再来向queue1队列提交作业:

    > set mapred.job.queue.name=queue1;
hive> select count(*) from t_aa_pc_log;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: User p_sdo_data_01 cannot perform operation SUBMIT_JOB on queue queue1.
 Please run "hadoop queue -showacls" command to find the queues you have access to .
    at org.apache.hadoop.mapred.ACLsManager.checkAccess(ACLsManager.java:179)
    at org.apache.hadoop.mapred.ACLsManager.checkAccess(ACLsManager.java:136)
    at org.apache.hadoop.mapred.ACLsManager.checkAccess(ACLsManager.java:113)
    at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3781)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)

作业提交失败!

最后， 可以使用 hadoop queue -showacls 命令查看队列信息:

[hadoop@localhost conf]$ hadoop queue -showacls
Queue acls for user :  hadoop

Queue  Operations
=====================
queue1  administer-jobs
queue2  submit-job,administer-jobs
queue3  submit-job,administer-jobs
queue4  submit-job,administer-jobs