hive的udf序列化报错(object not serializable)

报错如下:

Serialization stack:
	- object not serializable (class: java.lang.Object, value: java.lang.Object@1430d9d9)
	- field (class: com.yeahmobi.dmp.export.Hive2Es_user_bundle_country_many$$anonfun$run$2, name: nonLocalReturnKey1$1, type: class java.lang.Object)
	- object (class com.yeahmobi.dmp.export.Hive2Es_user_bundle_country_many$$anonfun$run$2, )
	- element of array (index: 4)
	- array (class [Ljava.lang.Object;, size 8)
	- field (class: org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10, name: references$1, type: class [Ljava.lang.Object;)
	- object (class org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10, )
	at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
	at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)
	at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
	at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:342)
	... 38 more

我的udf是:

ss.udf.register("getRouteCode",(routeKey:String) =>  {
  if(routeKey.isEmpty){
    return 99
  }
  (routeKey.hashCode() & Integer.MAX_VALUE) % 8
  
  
})

修改后的udf是

ss.udf.register("getRouteCode",(routeKey:String) =>  {
  val routeCode: Int =routeKey match  {
    case null|""=> 99
    case _=> {
      (routeKey.hashCode() & Integer.MAX_VALUE) % 8
    }
  }
  routeCode
})

错误就是 routeKey.isEmpty 这块的问题,如果要判空,就用模式匹配做,要不就会序列化不了

问题解决

你可能感兴趣的:(hive的udf序列化报错(object not serializable))