oozie5.0安装

环境:3台centos6.8集群,mysql8.0.12

下载oozie-5.0.0.tar.gz,解压到Documents/下,进入bin目录,编译:
bin/mkdistro.sh -Dhadoop.version=2.7.7  -DskipTests

bin/mkdistro.sh -Dhadoop.version=2.7.7 -Dhbase.version=1.2.0 -Dspark.version=2.1.0 -DskipTests

编译后的oozie-docs和oozie-webapp包拷贝到本地maven库中(分别位于docs和webapp目录下的target目录中),下面生成依赖包时用到.

编译完成后,生成依赖包库:
mvn dependency:copy-dependencies -DoutputDirectory=lib 
生成的依赖库保存到了oozie/zookeeper-security-tests/lib,下面初始化数据库调用脚本程序用到.

进入distro/target,看到oozie-5.0.0-distro.tar.gz文件,这是上面编译生成的文件,正是待安装部署的oozie包,解压到/usr/local/下
sudo tar zxvf oozie-5.0.0-distro.tar.gz -C /usr/local/
解压后的文件夹为oozie-5.0.0,更名为oozie,方便操作.
ln -s oozie-5.0.0 oozie

进入/usr/local/oozie/,新建目录libext.
mkdir libext
下载ext-2.2.zip到目录libext
下载mysql-connector-java-8.0.12.jar到目录libext
将hadoop下的文件拷贝到libext中
sudo cp /home/userb/Documents/hadoop-2.7.7/share/hadoop/*/*.jar /usr/local/oozie/libext/
sudo cp /home/userb/Documents/hadoop-2.7.7/share/hadoop/*/lib/*.jar /usr/local/oozie/libext/

创建hdfs目录:
hadoop fs -mkdir -p /user/oozie/share/lib
该目录即oozie-site.xml文件中的配置项:

     oozie.service.WorkflowAppService.system.libpath
     hdfs://192.168.44.130:9000/user/oozie/share/lib

上传jar到hdfs:
sudo -u userb bin/oozie-setup.sh sharelib create -fs hdfs://192.168.44.130:9000/user/oozie/share/lib -locallib oozie-sharelib-5.0.0.tar.gz
上传后自动解压

启动服务:
bin/oozied.sh start
界面显示
The SQL commands have been written to: /tmp/ooziedb-2877570196499241979.sql

Oozie server started - PID: 44238.

启动前端进程:
bin/oozied.sh run
查看日志:logs/oozie.log

查看状态:
bin/oozie admin -oozie http://192.168.44.130:11000/oozie -status
界面显示:SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
System mode: NORMAL

安装完成

示例:
[userb@master1 oozie]$ bin/oozie job -oozie http://192.168.44.130:11000/oozie -config examples/apps/map-reduce/job.properties -run
界面显示:
...
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
job: 0000000-190131214853903-oozie-user-W

[userb@master1 oozie]$ bin/oozie job -info 0000000-190131214853903-oozie-user-W
界面显示:
Workflow Name : map-reduce-wf
App Path      : hdfs://192.168.44.130:9000/user/oozie/examples/apps/map-reduce/workflow.xml
Status        : RUNNING
Run           : 0
User          : userb
Group         : -
Created       : 2019-01-31 16:20 GMT
Started       : 2019-01-31 16:20 GMT
Last Modified : 2019-01-31 16:20 GMT
Ended         : -
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code  
------------------------------------------------------------------------------------------------------------------------------------
0000000-190131214853903-oozie-user-W@mr-node                                  PREP      -                      -          -         
------------------------------------------------------------------------------------------------------------------------------------
0000000-190131214853903-oozie-user-W@:start:                                  OK        -                      OK         -  

问题:
http://192.168.44.130:11000/oozie/
报错:RawData:
org.apache.jasper.JasperException: java.lang.ClassNotFoundException: org.apache.jsp.index_jsp

该问题还没处理, 不过网上解决的方案一大堆, 有解决的朋友帮个忙, 谢谢!

关闭服务:
bin/oozied.sh stop

配置文件oozie-site.xml:

   
        oozie.service.ProxyUserService.proxyuser.userb.hosts
        *
       
            List of hosts the '#USER#' user is allowed to perform 'doAs'
            operations.

            The '#USER#' must be replaced with the username o the user who is
            allowed to perform 'doAs' operations.

            The value can be the '*' wildcard or a list of hostnames.

            For multiple users copy this property and replace the user name
            in the property name.
       
   

   
        oozie.service.ProxyUserService.proxyuser.userb.groups
        *
       
            List of groups the '#USER#' user is allowed to impersonate users
            from to perform 'doAs' operations.

            The '#USER#' must be replaced with the username o the user who is
            allowed to perform 'doAs' operations.

            The value can be the '*' wildcard or a list of groups.

            For multiple users copy this property and replace the user name
            in the property name.
       
   

   
        oozie.service.JPAService.create.db.schema
          false
       
   
        oozie.service.JPAService.jdbc.driver
        com.mysql.cj.jdbc.Driver
   
   
        oozie.service.JPAService.jdbc.url
        jdbc:mysql://192.168.44.130:3306/oozie?createDatabaseIfNotExist=true
   

   
        oozie.service.JPAService.jdbc.username
        oozie_user
   

   
        oozie.service.JPAService.jdbc.password
        hadoop
       
                DB user password.
                IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value,
                if empty Configuration assumes it is NULL.
       
   

   
        oozie.service.HadoopAccessorService.hadoop.configurations
        *=/home/userb/Documents/hadoop/etc/hadoop
   

     
        oozie.service.HadoopAccessorService.action.configurations
        *=/home/userb/Documents/hadoop/etc/hadoop
   

   
         oozie.service.SparkConfigurationService.spark.configurations
         *=/home/userb/Documents/spark/conf
   

   
   
         oozie.service.WorkflowAppService.system.libpath
         hdfs://192.168.44.130:9000/user/oozie/share/lib
   
   
        oozie.use.system.libpath
        true
       
                Default value of oozie.use.system.libpath. If user haven't specified =oozie.use.system.libpath=
                in the job.properties and this value is true and Oozie will include sharelib jars for workflow.
       
   

   
        oozie.subworkflow.classpath.inheritance
        true
   

不重启hadoop集群，而使配置生效:
hdfs dfsadmin -refreshSuperUserGroupsConfiguration
yarn rmadmin -refreshSuperUserGroupsConfiguration

log问题:
java.lang.ClassNotFoundException: org.apache.log4j.rolling.RollingFileAppender
解决:
下载apache-log4j-extras-1.2.17.jar
https://mvnrepository.com/artifact/log4j/apache-log4j-extras/1.2.17
该问题通过抽取项目依赖包解决:mvn dependency:copy-dependencies -DoutputDirectory=lib

初始化数据库(mysql8):
bin/ooziedb.sh create -sqlfile oozie.sql -run
生成oozie.sql文件,其中type = innodb应改为engine=innodb,因为type=InnoDB在5.0以前是可以使用的，但5.1之后就不行了。如果我们把type=InnoDB改为engine=InnoDB就不会有这个问题
修改脚本后,导入脚本
vi oozie.sql
:%s/TYPE = innodb/engine=innodb/g
:wq
然后导入脚本:mysql -uoozie_user -phadoop oozie < oozie.sql

问题:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.letsdobigdata.MaxTemperatureDriver.main(MaxTemperatureDriver.java:35)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 3 more

解决:
遇到这种错误后，判断jar包中有没有commons.cli的jar包，没有的话添加进去就好了
http://mirrors.hust.edu.cn/apache//commons/cli/binaries/commons-cli-1.4-bin.tar.gz

问题:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/openjpa/jdbc/meta/MappingTool

解决:
mvn dependency:copy-dependencies -DoutputDirectory=lib
maven项目所依赖的jar包都会复制到项目目录下的lib目录下

以上缺包少类问题,通过抽取maven依赖包解决,解决不了再下载; 上面的问题是安装过程中走过的弯路, 供参考!

 

参考:https://my.oschina.net/u/3825598/blog/1865904