Hadoop Oozie学习笔记(二) 使用OOZIE,通过命令行运行example

本博客属原创文章,转载请注明出处:http://guoyunsky.iteye.com/blog/1245092

欢迎加入Hadoop超级群: 180941958   

 

    Oozie下面有很多例子,提供测试.也可以将源码放入Eclipse中启动提交.这里就一起试下.但发现有些问题,一一解决吧.

     运行Oozie例子map-reduce,命令:

     $OOZIE_HOME/bin/oozie job -oozie http://localhost:11000/oozie -config /home/guoyun/hadoop/oozie-3.0.2/examples/apps/map-reduce/job.properties -run

     本以为会很顺利,但控制台报以下错误:

     Error: E0902 : E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]

    跑到日志中$OOZIE_HOME/logs/oozie.log发现有以下报错:

  2011-11-09 10:42:30,721 WARN V1JobsServlet:539 - USER[?] GROUP[users] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v1/jobs?action=start] error[E0902], E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]

org.apache.oozie.servlet.XServletException: E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
	at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:196)
	at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:89)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
	at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:281)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
	at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.oozie.service.AuthorizationException: E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
	at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:320)
	at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:185)
	... 16 more
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
	at org.apache.oozie.service.KerberosHadoopAccessorService.createFileSystem(KerberosHadoopAccessorService.java:208)
	at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:285)
	... 17 more
Caused by: java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:1131)
	at org.apache.hadoop.ipc.Client.call(Client.java:1107)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
	at $Proxy22.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:398)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:384)
	at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:111)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:213)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:180)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1514)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1548)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1530)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
	at org.apache.oozie.service.KerberosHadoopAccessorService$3.run(KerberosHadoopAccessorService.java:200)
	at org.apache.oozie.service.KerberosHadoopAccessorService$3.run(KerberosHadoopAccessorService.java:192)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
	at org.apache.oozie.service.KerberosHadoopAccessorService.createFileSystem(KerberosHadoopAccessorService.java:192)
	... 18 more
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:425)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:532)
	at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:210)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1244)
	at org.apache.hadoop.ipc.Client.call(Client.java:1075)
	... 37 more

  从Oozie的FAQ找到了解决办法:https://github.com/yahoo/oozie/wiki/FAQ,需要将oozie-default.xml中的 oozie.services属性拷贝到oozie-site.xml中,也就是以下属性:

     <property>
       <name>oozie.services</name>
        <value>
            org.apache.oozie.service.SchedulerService,
            org.apache.oozie.service.InstrumentationService,
            org.apache.oozie.service.CallableQueueService,
            org.apache.oozie.service.UUIDService,
            org.apache.oozie.service.ELService,
            org.apache.oozie.service.AuthorizationService,
            org.apache.oozie.service.KerberosHadoopAccessorService,
            org.apache.oozie.service.MemoryLocksService,
            org.apache.oozie.service.DagXLogInfoService,
            org.apache.oozie.service.SchemaService,
            org.apache.oozie.service.LiteWorkflowAppService,
            org.apache.oozie.service.JPAService,
            org.apache.oozie.service.StoreService,
            org.apache.oozie.service.CoordinatorStoreService,
            org.apache.oozie.service.SLAStoreService,
            org.apache.oozie.service.DBLiteWorkflowStoreService,
            org.apache.oozie.service.CallbackService,
            org.apache.oozie.service.ActionService,
            org.apache.oozie.service.ActionCheckerService,
            org.apache.oozie.service.RecoveryService,
            org.apache.oozie.service.PurgeService,
            org.apache.oozie.service.CoordinatorEngineService,
            org.apache.oozie.service.BundleEngineService,
            org.apache.oozie.service.DagEngineService,
            org.apache.oozie.service.CoordMaterializeTriggerService,
            org.apache.oozie.service.StatusTransitService,
            org.apache.oozie.service.PauseTransitService
        </value>
        <description>
            All services to be created and managed by Oozie Services singleton.
            Class names must be separated by commas.
        </description>
    </property>

   拷贝完成后重启OOZIE,继续跑map-reduce例子,控制台上却又报一样的错误:

   Error: E0902 : E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]

  还是看继续看日志吧,还是同样的错误.https://github.com/yahoo/oozie/wiki/FAQ中貌似也有解决办法,于是按照说明将org.apache.oozie.service.HadoopAccessorService 添加到oozie-size.xml中的oozie.services属性中.继续重启.

   还是一样的错误,怒了!难道不是Hadoop授权问题?而且本身来说我都是本地启动,不存在授权的问题啊.于是看下端口,lsof -i:8020,果真空空如也.早就应该想到这点了.于是去$HADOOP_HOME/conf/下查看core-site.xml发现HDFS的端口是9000,并不是这里默认的8020.同时看下maped-site.xml发现jobTracker端口是9001.于是去对应的app下,也就是我这里的$OOZIE_HOME/examples/apps/map-reduce下将job.properties中修改如下:

nameNode=hdfs://localhost:9000
jobTracker=localhost:9001
queueName=default
examplesRoot=hadoop/oozie-3.0.2/examples

oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/aggregator
outputDir=map-reduce

  于是继续运行这个example,报如下错误:

   Error: E0504 : E0504: App directory [hdfs://localhost:9000/user/guoyun/hadoop/oozie-3.0.2/examples/apps/aggregator] does not exist

  想想我现在是伪分布式下运行Hadoop,从HDFS中拿文件则需要先存入到HDFS中.还是直接调用本地文件方便调试,于是将job.properties修改如下:

nameNode=file://
jobTracker=localhost:9001
queueName=default
examplesRoot=hadoop/oozie-3.0.2/examples

oozie.wf.application.path=${nameNode}/home/${user.name}/${examplesRoot}/apps/map-reduce
outputDir=map-reduce

   再次运行得到以下结果:

   job: 0000001-111109111632287-oozie-guoy-W

   再去http://localhost:11000/oozie查看也能看到该job,并有相应的状态.

   OK!example运行成功!