使用Hue创建Spark1和Spark2的Oozie工作流

1.文档编写目的

---

使用Hue可以方便的通过界面制定Oozie的工作流，支持Hive、Pig、Spark、Java、Sqoop、MapReduce、Shell等等。Spark？那能不能支持Spark2的呢，接下来本文章就主要讲述如何使用Hue创建Spark1和Spark2的Oozie工作流。

• 内容概述

1.添加Spark2到Oozie的Share-lib

2.创建Spark2的Oozie工作流

3.创建Spark1的Oozie工作流

4.总结

• 测试环境

1.CM和CDH版本为5.11.2

2.操作系统：RedHat7.2

3.采用sudo权限的ec2-user用户操作

2.Oozie共享库添加Spark2

---

1.查看当前Oozie的share-lib共享库HDFS目录

ec2-user@ip-172-31-22-86jars$ oozie admin -ooziehttp://ip-172-31-22-86.ap-southeast-1.compute.internal:11000/oozie-sharelibupdate

ShareLib update status

host =http://ec2-54-169-0-36.ap-southeast-1.compute.amazonaws.com:11000/oozie

status = Successful

sharelibDirOld =hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424

sharelibDirNew =hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424

ec2-user@ip-172-31-22-86jars$

2.在Oozie的/user/oozie/share/lib/lib_20170921070424创建spark2目录

[ec2-user@ip-172-31-22-86 ~]$ sudo -u hdfs hadoop fs -mkdir /user/oozie/share/lib/lib_20170921070424/spark2

3.向spark2目录添加spark2的jars和oozie-sharelib-spark*.jar

[ec2-user@ip-172-31-22-86 jars]$ pwd
/opt/cloudera/parcels/SPARK2/lib/spark2/jars
[ec2-user@ip-172-31-22-86 jars]$ sudo -u hdfs hadoop fs -put *.jar /user/oozie/share/lib/lib_20170921070424/spark2

[ec2-user@ip-172-31-22-86 spark]$ pwd
/opt/cloudera/parcels/CDH/lib/oozie/oozie-sharelib-yarn/lib/spark
[ec2-user@ip-172-31-22-86 spark]$ sudo -u hdfs hadoop fs -put oozie-sharelib-spark*.jar /user/oozie/share/lib/lib_20170921070424/spark2

4.修改目录属主及权限

[ec2-user@ip-172-31-22-86 ~]$ sudo -u hdfs hadoop fs –chown -R oozie:oozie /user/oozie/share/lib/lib_20170921070424/spark2
[ec2-user@ip-172-31-22-86 ~]$ sudo -u hdfs hadoop fs –chmod -R 775 /user/oozie/share/lib/lib_20170921070424/spark2

5.更新Oozie的share-lib

[ec2-user@ip-172-31-22-86 spark]$ oozie admin -oozie http://ip-172-31-22-86.ap-southeast-1.compute.internal:11000/oozie -sharelibupdate
[ShareLib update status]
        host = http://ec2-54-169-0-36.ap-southeast-1.compute.amazonaws.com:11000/oozie
        status = Successful
        sharelibDirOld = hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424
        sharelibDirNew = hdfs://ip-172-31-21-45.ap-southeast-1.compute.internal:8020/user/oozie/share/lib/lib_20170921070424

[ec2-user@ip-172-31-22-86 spark]$ 

6.确认spark2已经添加到共享库

[ec2-user@ip-172-31-22-86 spark]$ oozie admin -oozie http://ip-172-31-22-86.ap-southeast-1.compute.internal:11000/oozie -shareliblist
[Available ShareLib]
spark2
oozie
hive
distcp
hcatalog
sqoop
mapreduce-streaming
spark
hive2
pig
[ec2-user@ip-172-31-22-86 spark]$ 

3.创建Spark2的Oozie工作流

---

1.登录Hue，创建Oozie工作流

2.进入WorkSpace

点击lib

在命令行将Spark2自带的example例子上传到/user/hue/oozie/workspaces/hue-oozie-1507832616.28/lib目录

[ec2-user@ip-172-31-22-86 jars]$ pwd
/opt/cloudera/parcels/SPARK2/lib/spark2/examples/jars
[ec2-user@ip-172-31-22-86 jars]$ sudo -u hdfs hadoop fs -put spark-examples_2.11-2.1.0.cloudera1.jar /user/hue/oozie/workspaces/hue-oozie-1507832616.28/lib
[ec2-user@ip-172-31-22-86 jars]$ 

3.添加Spark2任务

设置使用Spark2，否则默认使用的Spark1

完成配置，点击保存

4.保存完成后，点击运行测试是否正常

运行成功

4.创建Spark1的Oozie工作流

---

1.创建Oozie工作流

进入WorkSpace

2.将Spark1的example的jar包上传至该WorkSpace的lib目录

[ec2-user@ip-172-31-22-86 lib]$ pwd
/opt/cloudera/parcels/CDH/lib/spark/examples/lib
[ec2-user@ip-172-31-22-86 lib]$ sudo -u hdfs hadoop fs -put spark-examples-1.6.0-cdh5.11.2-hadoop2.6.0-cdh5.11.2.jar /user/hue/oozie/workspaces/hue-oozie-1507860705.24/lib
[ec2-user@ip-172-31-22-86 lib]$ 

3.添加Spark1的任务到Oozie

4.保存Oozie，然后点击提交

运行成功

5.常见问题

---

1.在使用Hue创建Spark2的Oozie工作流时运行异常

2017-10-16 23:20:07,086 WARN org.apache.oozie.action.hadoop.SparkActionExecutor: SERVER[ec2-54-179-152-169.ap-southeast-1.compute.amazonaws.com] USER[admin] GROUP[-] TOKEN[] APP[MyFirstSpark2] JOB[0000000-171016230402705-oozie-oozi-W] ACTION[0000000-171016230402705-oozie-oozi-W@spark-1411] Launcher exception: Exception when registering SparkListener
org.apache.spark.SparkException: Exception when registering SparkListener
   at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2193)
   at org.apache.spark.SparkContext.<init>(SparkContext.scala:562)
   at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
   at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
   at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
   at scala.Option.getOrElse(Option.scala:121)
   at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
   at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
   at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
   at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
   at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
   at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
   at org.apache.oozie.action.hadoop.SparkMain.runSpark(SparkMain.java:178)
   at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:90)
   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
   at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:57)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: com.cloudera.spark.lineage.ClouderaNavigatorListener
   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:270)
   at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
   at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2159)
   at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2156)
   at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
   at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
   at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2156)
   ... 34 more

解决方法：

保存配置然后重启相应服务。

6.总结

---

使用Oozie创建Spark的工作流，如果需要运行Spark2的作业则需要向Oozie的Share-lib库中添加Spark2的支持，并在创建Spark2作业的时候需要指定Share-lib为spark2（Oozie默认的支持Spark1），否则会运行失败报Spark2的类找不到异常。