Java在网络通信中应该如何选择合适的序列化框架?
前言
说到网络通信就会牵涉到数据的序列化与反序列化，现如今序列化框架也是层出不穷，比如FST、Kryo、ProtoBuffer、Thrift、Hessian、Avro、MsgPack等等，有的人可能会有疑问，为什么市面上有这么多框架，JDK不是已经有自带的Serializable序列化接口吗?很遗憾地说出这个事实，作为JDK自带地序列化机制，无论是在时间还是空间上的性能不尽人意，但凡时间或者空间上性能优越一点，也不至于让人诟病这么久。所以也就出现了这么多序列化框架，另外即便JDK序列化可以实现，但是无法在跨语言的网络通信中表现出色，除非在多个语言端各自定义好每一个对象，但是这样做无疑效率是最低的，需要提前定义的对象太多了，人力明显不够用。所以也就出现了以ProtoBuffer、Kryo、Thrift等支持跨语言的框架，接下来让我们看下各个框架之间的比较，有比较才能在业务中有更多的选择
| 序列化框架 | 通用性 |
|---|---|
| JDK Serializer | 只适用于Java |
| FST | 只适用于Java |
| Kryo | 主要适用于Java(可复杂支持多种语言) |
| Protocol buffer | 支持多种语言 |
| Thrift | 支持多种语言 |
| Hessian | 支持多种语言 |
| Avro | 支持多种语言 |
| MsgPack | 支持多种语言 |
性能比较
从时间和空间复杂度上来说，kryo的表现是其他序列化框架当中比较好的，序列化之后的大小空间上以及序列化/反序列化时间都不错，但是它没有那么直接不支持跨语言。
接下来我们再来看看针对大小数据序列化的结果，数据的大小也会影响序列化的结果。
分析结果
如果你的系统架构设计中设计到了多语言，那么Proto Buffer和MsgPack，avro将会是个不错的选择，当你的系统是Java时，还可以考虑kryo，它的序列化和反序列化时间相对均衡。
使用示例
<dependencies>
<dependency>
<groupId>io.nettygroupId>
<artifactId>netty-allartifactId>
<version>4.1.42.Finalversion>
dependency>
<dependency>
<groupId>org.msgpackgroupId>
<artifactId>msgpackartifactId>
<version>0.6.12version>
dependency>
<dependency>
<groupId>org.slf4jgroupId>
<artifactId>slf4j-apiartifactId>
<version>1.7.30version>
<scope>compilescope>
dependency>
<dependency>
<groupId>ch.qos.logbackgroupId>
<artifactId>logback-coreartifactId>
<version>1.2.4version>
dependency>
<dependency>
<groupId>ch.qos.logbackgroupId>
<artifactId>logback-classicartifactId>
<version>1.2.4version>
dependency>
<dependency>
<groupId>com.itextpdfgroupId>
<artifactId>itextpdfartifactId>
<version>5.5.8version>
dependency>
<dependency>
<groupId>org.bouncycastlegroupId>
<artifactId>bcprov-jdk15onartifactId>
<version>1.49version>
<type>jartype>
<scope>compilescope>
<optional>trueoptional>
dependency>
<dependency>
<groupId>org.bouncycastlegroupId>
<artifactId>bcpkix-jdk15onartifactId>
<version>1.49version>
<type>jartype>
<scope>compilescope>
<optional>trueoptional>
dependency>
<dependency>
<groupId>de.javakaffeegroupId>
<artifactId>kryo-serializersartifactId>
<version>0.42version>
dependency>
<dependency>
<groupId>junitgroupId>
<artifactId>junitartifactId>
<version>4.13.1version>
<scope>testscope>
dependency>
<dependency>
<groupId>com.google.protobufgroupId>
<artifactId>protobuf-javaartifactId>
<version>2.6.1version>
dependency>
dependencies>
1.ProtoBuffer
Proto文件如何生成JavaProto对象?这篇博客已经介绍过了ProtoBuffer了，这里不再赘述，有兴趣的小伙伴可以看看
2.Kryo
实体类
package serializable.protogenesis;
import org.msgpack.annotation.Message;
import java.io.Serializable;
import java.nio.ByteBuffer;
@Message
public class UserInfo implements Serializable {
/**
* 默认序列号
*/
private static final long serialVersionUID = 7627113094707995002L;
private String userName;
private int userID;
public String getUserName() {
return userName;
}
public UserInfo setUserName(String userName) {
this.userName = userName;
return this;
}
public int getUserID() {
return userID;
}
public UserInfo setUserID(int userID) {
this.userID = userID;
return this;
}
// 自行序列化
public byte[] codeC() {
ByteBuffer buffer = ByteBuffer.allocate(1024);
// userName转换为字节数组value
byte[] value = this.userName.getBytes();
// 写入字节数组value的长度
buffer.putInt(value.length);
// 写入字节数组value的值
buffer.put(value);
// 写入userID的值
buffer.putInt(this.userID);
// 准备读取buffer中的数据
buffer.flip();
value = null;
byte[] result = new byte[buffer.remaining()];
// buffer中的数据写入字节数组并作为结果返回
buffer.get(result);
return result;
}
// 自行序列化方法2
public byte[] codeC(ByteBuffer buffer) {
buffer.clear();
byte[] value = this.userName.getBytes();
buffer.putInt(value.length);
buffer.put(value);
buffer.putInt(this.userID);
buffer.flip();
value = null;
byte[] result = new byte[buffer.remaining()];
buffer.get(result);
return result;
}
@Override
public String toString() {
return "UserInfo{" +
"userName='" + userName + '\'' +
", userID=" + userID +
'}';
}
}
KryoSerializer
package adv.kryocodec;
import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.io.Input;
import com.esotericsoftware.kryo.io.Output;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
public class KryoSerializer {
private static Kryo kryo = KryoFactory.createKryo();
// 序列化
public static void serialize(Object object, ByteBuf out) {
long start = System.currentTimeMillis();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
Output output = new Output(baos);
kryo.writeClassAndObject(output, object);
output.flush();
output.close();
byte[] b = baos.toByteArray();
try {
baos.flush();
baos.close();
} catch (IOException e) {
e.printStackTrace();
}
out.writeBytes(b);
long end = System.currentTimeMillis();
System.out.println("The Kryo serializable length is "+ b.length +", serialize time is :" + (end - start));
}
// 序列化为一个字节数组，主要用在消息摘要上
public static byte[] obj2Bytes(Object object) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
Output output = new Output(baos);
kryo.writeClassAndObject(output, object);
output.flush();
output.close();
byte[] b = baos.toByteArray();
try {
baos.flush();
baos.close();
} catch (IOException e) {
e.printStackTrace();
}
return b;
}
public static Object deserialize(ByteBuf out) {
if (out == null) {
return null;
}
Input input = new Input(new ByteBufInputStream(out));
return kryo.readClassAndObject(input);
}
}
KryoFactory
在这个工厂中我们提前注册了很多序列化器，这样会加快我们序列化的速度
package adv.kryocodec;
import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.serializers.DefaultSerializers;
import de.javakaffee.kryoserializers.*;
import java.lang.reflect.InvocationHandler;
import java.math.BigDecimal;
import java.math.BigInteger;
import java.net.URI;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.regex.Pattern;
public class KryoFactory {
public static Kryo createKryo() {
Kryo kryo = new Kryo();
kryo.setRegistrationRequired(false);
kryo.register(Arrays.asList("").getClass(), new ArraysAsListSerializer());
kryo.register(GregorianCalendar.class, new GregorianCalendarSerializer());
kryo.register(InvocationHandler.class, new JdkProxySerializer());
kryo.register(BigDecimal.class, new DefaultSerializers.BigDecimalSerializer());
kryo.register(BigInteger.class, new DefaultSerializers.BigIntegerSerializer());
kryo.register(Pattern.class, new RegexSerializer());
kryo.register(BitSet.class, new BitSetSerializer());
kryo.register(URI.class, new URISerializer());
kryo.register(UUID.class, new UUIDSerializer());
UnmodifiableCollectionsSerializer.registerSerializers(kryo);
SynchronizedCollectionsSerializer.registerSerializers(kryo);
kryo.register(HashMap.class);
kryo.register(ArrayList.class);
kryo.register(LinkedList.class);
kryo.register(HashSet.class);
kryo.register(TreeSet.class);
kryo.register(Hashtable.class);
kryo.register(Date.class);
kryo.register(Calendar.class);
kryo.register(ConcurrentHashMap.class);
kryo.register(SimpleDateFormat.class);
kryo.register(GregorianCalendar.class);
kryo.register(Vector.class);
kryo.register(BitSet.class);
kryo.register(StringBuffer.class);
kryo.register(StringBuilder.class);
kryo.register(Object.class);
kryo.register(Object[].class);
kryo.register(String[].class);
kryo.register(byte[].class);
kryo.register(char[].class);
kryo.register(int[].class);
kryo.register(float[].class);
kryo.register(double[].class);
return kryo;
}
}
TestPerform
package adv.kryocodec;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;
import serializable.protogenesis.UserInfo;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
public class TestPerform {
public static void main(String[] args) throws IOException {
UserInfo info = new UserInfo();
info.setUserID(100).setUserName("Hello World");
long start = System.currentTimeMillis();
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(bos);
os.writeObject(info);
os.flush();
os.close();
byte[] b = bos.toByteArray();
long end = System.currentTimeMillis();
System.out.println("The JDK serializable length is :" + b.length + ", time is :" + (end - start));
ByteBuf sendBuf = Unpooled.buffer();
KryoSerializer.serialize(info, sendBuf);
UserInfo deserialize = (UserInfo)KryoSerializer.deserialize(sendBuf);
System.out.println("deserialize = " + deserialize.toString());
}
}
3.MsgPack
TestMsgPackPerform
package serializable.msgpack;
import org.msgpack.MessagePack;
import serializable.protogenesis.UserInfo;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
public class TestMsgPackPerform {
public static void main(String[] args) throws IOException {
UserInfo info = new UserInfo();
info.setUserID(100).setUserName("Hello World");
long start = System.currentTimeMillis();
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(bos);
os.writeObject(info);
os.flush();
os.close();
byte[] b = bos.toByteArray();
long end = System.currentTimeMillis();
System.out.println("The JDK serializable length is :" + b.length + ", time is :" + (end - start));
MessagePack messagePack = new MessagePack();
byte[] bytes = messagePack.write(info);
UserInfo read = messagePack.read(bytes, UserInfo.class);
System.out.println("The MsgPack serializable length is :" + bytes.length);
System.out.println("read = " + read);
}
}
由于在序列化时会申请ByteBuf来操作，所以这个申请内存(无论是堆内存还是直接内存)都是需要耗费点时间的，所以不能把这个时间也算进去，这样我们得到的时间是不准的,只要在大量的序列化情况下才能看出效果。另外注意在实体类上加@Message这个注解，否则会报错
附录：NIO序列化
我们都知道NIO中也有ByteBuffer可以用来做序列化
TestUserInfo
public class TestUserInfo {
public static void main(String[] args) throws IOException {
UserInfo info = new UserInfo();
info.setUserID(100).setUserName("Hello World");
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream os = new ObjectOutputStream(bos);
os.writeObject(info);
os.flush();
os.close();
byte[] b = bos.toByteArray();
System.out.println("The JDK serializable length is :" + b.length);
bos.close();
System.out.println("-------------------------------------------");
System.out.println("the byte array serializable length is :" + info.codeC().length);
}
}
PerformTestUserInfo
package serializable.protogenesis;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.nio.ByteBuffer;
public class PerformTestUserInfo {
public static void main(String[] args) throws IOException {
UserInfo info = new UserInfo();
info.setUserID(100).setUserName("Hello World");
int loop = 1000000;
ByteArrayOutputStream bos = null;
ObjectOutputStream os = null;
long startTime = System.currentTimeMillis();
for (int i = 0; i < loop; i++) {
bos = new ByteArrayOutputStream();
os = new ObjectOutputStream(bos);
os.writeObject(info);
os.flush();
os.close();
byte[] bytes = bos.toByteArray();
bos.close();
}
long endTime = System.currentTimeMillis();
System.out.println("The JDK serializable cost time is :" + (endTime - startTime) + "ms");
System.out.println("---------------------------------");
ByteBuffer buffer = ByteBuffer.allocate(1024);
startTime = System.currentTimeMillis();
for (int i = 0; i < loop; i++) {
byte[] bytes = info.codeC(buffer);
}
endTime = System.currentTimeMillis();
System.out.println("The byte array serializable cost time is :" + (endTime - startTime) + "ms");
}
}
NIO原生的序列化机制也有很不错的机制