hadoop mapred-queue-acls 配置

hadoop作业提交时可以指定相应的队列,例如:-Dmapred.job.queue.name=queue2

通过对mapred-queue-acls.xml和mapred-site.xml配置可以对不同的队列实现不同用户的提交权限.
先编辑mapred-site.xml,修改配置如下(增加四个队列):

 
 
  1. <property>
  2. <name>mapred.queue.names</name>
  3. <value>default,queue1,queue2,queue3,queue4</value>
  4. <description>Commaseparatedlistofqueuesconfiguredforthisjobtracker.
  5. Jobsareaddedtoqueuesandschedulerscanconfiguredifferent
  6. schedulingpropertiesforthevariousqueues.Toconfigureaproperty
  7. foraqueue,thenameofthequeuemustmatchthenamespecifiedinthis
  8. value.Queuepropertiesthatarecommontoallschedulersareconfigured
  9. herewiththenamingconvention,mapred.queue.$QUEUE-NAME.$PROPERTY-NAME,
  10. fore.g.mapred.queue.default.submit-job-acl.
  11. Thenumberofqueuesconfiguredinthisparametercoulddependonthe
  12. typeofschedulerbeingused,asspecifiedin
  13. mapred.jobtracker.taskScheduler.Forexample,theJobQueueTaskScheduler
  14. supportsonlyasinglequeue,whichisthedefaultconfiguredhere.
  15. Beforeaddingmorequeues,ensurethatthescheduleryou'veconfigured
  16. supportsmultiplequeues.
  17. </description>
  18. </property>

修改生效后通过jobtrack界面可以看到配置的队列信息:

hadoop mapred-queue-acls 配置

要对队列进行控制, 还需要编辑mapred-queue-acls.xml文件

 
 
  1. <property>
  2. <name>mapred.queue.queue1.acl-submit-job</name>
  3. <value>''</value>
  4. <description>Commaseparatedlistofuserandgroupnamesthatareallowed
  5. tosubmitjobstothe'default'queue.Theuserlistandthegrouplist
  6. areseparatedbyablank.Fore.g.user1,user2group1,group2.
  7. Ifsettothespecialvalue'*',itmeansallusersareallowedto
  8. submitjobs.Ifsetto''(i.e.space),nouserwillbeallowedtosubmit
  9. jobs.
  10. ItisonlyusedifauthorizationisenabledinMap/Reducebysettingthe
  11. configurationpropertymapred.acls.enabledtotrue.
  12. IrrespectiveofthisACLconfiguration,theuserwhostartedtheclusterand
  13. clusteradministratorsconfiguredvia
  14. mapreduce.cluster.administratorscansubmitjobs.
  15. </description>
  16. </property>

要配置多个队列, 只需要重复添加上面配置信息,修改队列名称和value值,为方便测试,queue1禁止所有用户向其提交作业.
要使该配置生效, 还需要修改mapred-site.xml,将mapred.acls.enabled值设置为true

 
 
  1. <property>
  2. <name>mapred.acls.enabled</name>
  3. <value>true</value>
  4. <description>SpecifieswhetherACLsshouldbechecked
  5. forauthorizationofusersfordoingvariousqueueandjobleveloperations.
  6. ACLsaredisabledbydefault.Ifenabled,accesscontrolchecksaremadeby
  7. JobTrackerandTaskTrackerwhenrequestsaremadebyusersforqueue
  8. operationslikesubmitjobtoaqueueandkillajobinthequeueandjob
  9. operationslikeviewingthejob-details(Seemapreduce.job.acl-view-job)
  10. orformodifyingthejob(Seemapreduce.job.acl-modify-job)using
  11. Map/ReduceAPIs,RPCsorviatheconsoleandwebuserinterfaces.
  12. </description>
  13. </property>

重启hadoop, 使配置生效, 接下来拿hive进行测试:
先使用queue2队列:

 
 
  1. setmapred.job.queue.name=queue2;
  2. hive>
  3. >selectcount(*)fromt_aa_pc_log;
  4. TotalMapReducejobs=1
  5. LaunchingJob1outof1
  6. Numberofreducetasksdeterminedatcompiletime:1
  7. Inordertochangetheaverageloadforareducer(inbytes):
  8. sethive.exec.reducers.bytes.per.reducer=<number>
  9. Inordertolimitthemaximumnumberofreducers:
  10. sethive.exec.reducers.max=<number>
  11. Inordertosetaconstantnumberofreducers:
  12. setmapred.reduce.tasks=<number>
  13. StartingJob=job_201205211843_0002,TrackingURL=http://192.168.189.128:50030/jobdetails.jsp?jobid=job_201205211843_0002
  14. KillCommand=/opt/app/hadoop-0.20.2-cdh3u3/bin/hadoopjob-Dmapred.job.tracker=192.168.189.128:9020-killjob_201205211843_0002
  15. 2012-05-2118:45:01,593Stage-1map=0%,reduce=0%
  16. 2012-05-2118:45:04,613Stage-1map=100%,reduce=0%
  17. 2012-05-2118:45:12,695Stage-1map=100%,reduce=100%
  18. EndedJob=job_201205211843_0002
  19. OK
  20. 136003
  21. Timetaken:14.674seconds
  22. hive>

作业成功完成

再来向queue1队列提交作业:

 
 
  1. >setmapred.job.queue.name=queue1;
  2. hive>selectcount(*)fromt_aa_pc_log;
  3. TotalMapReducejobs=1
  4. LaunchingJob1outof1
  5. Numberofreducetasksdeterminedatcompiletime:1
  6. Inordertochangetheaverageloadforareducer(inbytes):
  7. sethive.exec.reducers.bytes.per.reducer=<number>
  8. Inordertolimitthemaximumnumberofreducers:
  9. sethive.exec.reducers.max=<number>
  10. Inordertosetaconstantnumberofreducers:
  11. setmapred.reduce.tasks=<number>
  12. org.apache.hadoop.ipc.RemoteException:org.apache.hadoop.security.AccessControlException:Userp_sdo_data_01cannotperformoperationSUBMIT_JOBonqueuequeue1.
  13. Pleaserun"hadoopqueue-showacls"commandtofindthequeuesyouhaveaccessto.
  14. atorg.apache.hadoop.mapred.ACLsManager.checkAccess(ACLsManager.java:179)
  15. atorg.apache.hadoop.mapred.ACLsManager.checkAccess(ACLsManager.java:136)
  16. atorg.apache.hadoop.mapred.ACLsManager.checkAccess(ACLsManager.java:113)
  17. atorg.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3781)
  18. atsun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)
  19. atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  20. atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  21. atjava.lang.reflect.Method.invoke(Method.java:597)
  22. atorg.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
  23. atorg.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
  24. atorg.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
  25. atjava.security.AccessController.doPrivileged(NativeMethod)
  26. atjavax.security.auth.Subject.doAs(Subject.java:396)
  27. atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
  28. atorg.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)

作业提交失败!

最后, 可以使用hadoop queue -showacls 命令查看队列信息:

 
 
  1. [hadoop@localhostconf]$hadoopqueue-showacls
  2. Queueaclsforuser:hadoop
  3. QueueOperations
  4. =====================
  5. queue1administer-jobs
  6. queue2submit-job,administer-jobs
  7. queue3submit-job,administer-jobs
  8. queue4submit-job,administer-jobs

你可能感兴趣的:(hadoop)