Spark Serializable与KryoRegistrator比较

一、示例代码

        List<HashMap<String, Object>> mapList = buildMapArray();

        List<Tuple2<String, Iterable<HashMap<String, Object>>>> collect = javaSparkContext.parallelize(mapList).mapToPair(new PairFunction<HashMap<String, Object>, String, HashMap<String, Object>>() {
            @Override
            public Tuple2<String, HashMap<String, Object>> call(HashMap<String, Object> student) throws Exception {
                return new Tuple2<String, HashMap<String, Object>>("x", student);
            }
            //会产生shuffle
        }).groupByKey().collect();

        LOG.info("collect : " + collect);

二、执行结果：看read和write，这是Serializable的结果

三、使用KryoRegistrator需要注意事项

   public void registerClasses(Kryo kryo) {
        //kryo.register(SparkSerilizeTest.Student.class);
        //kryo.register(SparkSerilizeTest.class);
        kryo.register(java.util.HashMap.class);
        //kryo.register(java.util.List.class);
    }

sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
sparkConf.set("spark.kryo.registrator", "com.sdyc.ndmp.wechat.job.MyRegistrator");

四、执行结果：看read和write，这是KryoRegistrator结果

所以，序列化这块能提高不少效率