HDP 2.6 上配置 Oozie 同时支持 Spark 1.6 和 2.1 两个版本

需求

历史任务基于 Spark1. 新任务计划转移到 Spark2. 需要 Oozie 同时支持两个版本.

步骤

1 配置 sharelib

参照 Hortonworks 文档 创建 spark2 sharelib:

hdfs dfs -mkdir /user/oozie/share/lib/lib_/spark2
hdfs dfs -put \
    /usr/hdp//spark2/jars/* \
    /user/oozie/share/lib/lib_/spark2/
hdfs dfs -cp \
    /user/oozie/share/lib/lib_/spark/oozie-sharelib-spark-.jar \
    /user/oozie/share/lib/lib_/spark2/
hdfs dfs -cp \
    /user/oozie/share/lib/lib_/spark/hive-site.xml \
    /user/oozie/share/lib/lib_/spark2/
hdfs dfs -put \
    /usr/hdp//spark2/python/lib/py* \
    /user/oozie/share/lib/lib_/spark2/
oozie admin –sharelibupdate

检查是否创建成功:

oozie admin –shareliblist spark2

2 配置 job.properties

设置以下属性值:

oozie.action.sharelib.for.spark=spark2

2 删除重复的 jar 包

完成上述配置后运行 workflow 会报重复加载文件的错误:

WARN SparkActionExecutor:523 - SERVER[] USER[admin] GROUP[-] TOKEN[] APP[Workflow2] JOB[0000012-170717153234639-oozie-oozi-W] ACTION[0000012-170717153234639-oozie-oozi-W@spark_1] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Attempt to add (hdfs://:8020/user/oozie/share/lib/lib_20170613110051/oozie/aws-java-sdk-core-1.10.6.jar) multiple times to the distributed cache.
2017-07-19 12:36:53,275 WARN SparkActionExecutor:523 - SERVER[] USER[admin] GROUP[-] TOKEN[] APP[Workflow2] JOB[0000012-170717153234639-oozie-oozi-W] ACTION[0000012-170717153234639-oozie-oozi-W@spark_1] Launcher exception: Attempt to add (hdfs://:8020/user/oozie/share/lib/lib_20170613110051/oozie/aws-java-sdk-core-1.10.6.jar) multiple times to the distributed cache.
java.lang.IllegalArgumentException: Attempt to add (hdfs://:8020/user/oozie/share/lib/lib_20170613110051/oozie/aws-java-sdk-core-1.10.6.jar) multiple times to the distributed cache.

原因

/user/oozie/share/lib/lib_/oozie/ 和 /user/oozie/share/lib/lib_/spark2/ 下存在同名 jar 包.

解决方法

删除 spark2(尽量减少对其他非 spark2 任务的影响) 下的同名 jar 包.

3 新旧版本的 jar 包兼容

完成上述配置后运行 workflow 会报以下错误:

[0000591-171111105051439-oozie-oozi-W] ACTION[0000591-171111105051439-oozie-oozi-W@spark-b645] Launcher exception: com.fasterxml.jackson.databind.JavaType.isReferenceType()Z
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JavaType.isReferenceType()Z
at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByLookup(BasicSerializerFactory.java:302)

原因

spark2 加载了 /user/oozie/share/lib/lib_/oozie/ 下 版本过旧的 jar 包:

jackson-annotations-2.4.0.jar (?)
oozie/jackson-core-2.4.4.jar
jackson-databind-2.4.4.jar

解决方法

由于以上版本的 jar 被Spark1任务依赖, 不能直接更新或删除. 因为这个 Oozie 示例只运行 spark action, 简单地把这些
jar move() 到 /user/oozie/share/lib/lib_/spark/ 目录即可.

此时, spark1 任务加载的是 /user/oozie/share/lib/lib_/spark/ 下的

jackson-annotations-2.4.0.jar
oozie/jackson-core-2.4.4.jar
jackson-databind-2.4.4.jar

此时, spark2 任务加载的是 /user/oozie/share/lib/lib_/spark2/ 下的

jackson-annotations-2.6.5.jar
oozie/jackson-core-2.6.5.jar
jackson-databind-2.6.5.jar

其他解决方法

hortonworks 有提到可以重新生成oozie sharelib:

/usr/hdp/2.3.2.0-2950/oozie/bin/oozie-setup.sh sharelib create -locallib /usr/hdp//oozie/oozie-sharelib.tar.gz -fs hdfs://:8020
oozie admin -oozie http://:11000/oozie -sharelibupdate

DONE