sprak 序列化相关错误

在提交spark任务的过程中，如果设置了使用kryo进行序列化，即:

conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")

可能会遇到以下异常：

com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
Serialization trace:
location_ (knowledge.pub.Properties$AccessPlayControl$RegionControl)
regionControl_ (knowledge.pub.Properties$AccessPlayControl)
accessPlayControl_ (knowledge.pub.Knowledge$Episode)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
	at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:127)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
	at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:127)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)

原因是因为kryo早期的版本中，对immutable list支持的不太好，这个时候需要引入第三方包：

<dependency>
    <groupId>de.javakaffeegroupId>
    <artifactId>kryo-serializersartifactId>
    <version>0.42version>
dependency>

然后自定义注册类：

import com.esotericsoftware.kryo.Kryo;
import de.javakaffee.kryoserializers.protobuf.ProtobufSerializer;
import org.apache.spark.serializer.KryoRegistrator;

public class MyKryoRegistrator implements KryoRegistrator {
    @Override
    public void registerClasses(Kryo kryo) {
        // Probably should use proto serializer for your proto classes
        kryo.register( Knowledge.Episode.class, new ProtobufSerializer() );

    }
}

然后在conf中注册这个类：

conf.set("spark.kryo.registrator", "com.iqiyi.lego.job.beehive.spark.registrator.MyKryoRegistrator");

可以解决这个问题。