protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较

公司一个cs架构项目接口,使用json传输数据大小接近10M,简单的研究了一下protobuf 、protostuff,以及针对这个接口场景 使用protobuf 、protostuff、json序列化nginx的gzip压缩传输后文件大小的比较。
1.protobuf简单使用
下载protoc-3.5.1-win32.zip 和 protobuf-java-3.5.1.zip 分别解压
把 protoc-3.5.1-win32.zip里的protoc.exe拷贝到 E:\develop\protobuf-3.5.1\src (不考也行)
E:\develop\protobuf-3.5.1\src\protoc.exe 是protoc.exe 所在目录
E:\develop\protobuf-3.5.1\examples 是 .proto所在的目录
进入要编译的.proto文件目录(E:\develop\protobuf-3.5.1\examples) 执行一下命令
E:\develop\protobuf-3.5.1\examples>E:\develop\protobuf-3.5.1\src\protoc.exe --j
va_out=. addressbook.proto
会在当前目录生成AddressBookProtos.java
protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较_第1张图片
还可使用maven插件使每次打包的时候自动把.proto文件编译成.java文件
这里没有这么做,使用protocol buffer主要的工作是要另外编写.proto文件
对应现有系统改的比较大所以没有使用protobuffer方案。
.proto还是比较简单,可以自行学习
下面给出一下例子,只是为了忘记时候可以回忆一下,可以略过
protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较_第2张图片
syntax = "proto3";
option java_package = "com.*.*.*.*.rcp.protobuf";
option java_outer_classname = "XXXXProto";
import public "FltBookResult.proto";
import "ResultBase.proto";
message XXXXInfo {
/**
*

* Description:航段唯一 ID 号
*

*/
string soflSeqNr = 1;
.......
FltBookResult fltBookResult = 96;
ResultBase resultBase = 97;
}
2.prostostuff的使用
结合springmvc的 Converter
import com.google.common.base.Stopwatch;
import io.protostuff.LinkedBuffer;
import io.protostuff.ProtobufIOUtil;
import io.protostuff.Schema;
import io.protostuff.runtime.RuntimeSchema;
import net.jpountz.lz4.LZ4BlockOutputStream;
import org.apache.commons.io.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpInputMessage;
import org.springframework.http.HttpOutputMessage;
import org.springframework.http.MediaType;
import org.springframework.http.converter.AbstractHttpMessageConverter;
import org.springframework.http.converter.HttpMessageNotReadableException;
import org.springframework.http.converter.HttpMessageNotWritableException;
import org.xerial.snappy.SnappyOutputStream;
 
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.charset.Charset;
import java.util.zip.GZIPOutputStream;
 
/**
*

* An { @code HttpMessageConverter} that can read and write
* Google Protocol Buffer (Protobuf) messages using the
* open-source Protostuff library. Its advantage over native Protobuf
* serialization and deserialization is that it can work with normal { @code POJO}s, as compared to the native
* implementation that requires the objects to implement the { @code Message} interface from the Protobuf Java library.
* This allows applications to use Protobuf with existing classes instead of having to re-generate them using the
* Protobuf compiler.
*

*

* Supports the { @code application/x-protobuf} media type. Regular Spring MVC application clients can use this as the
* media type for the { @code Accept} and { @code Content-Type} HTTP headers for exchanging information as Protobuf
* messages.
*

*/
public class ProtostuffHttpMessageConverter extends AbstractHttpMessageConverter {
private Logger logger = LoggerFactory. getLogger (getClass());
public static final Charset DEFAULT_CHARSET = Charset. forName ("UTF-8");
public static final MediaType MEDIA_TYPE = new MediaType("application", "x-protobuf", DEFAULT_CHARSET );
public static final MediaType MEDIA_TYPE_LZ4 = new MediaType("application", "x-protobuf-lz4", DEFAULT_CHARSET );
public static final MediaType MEDIA_TYPE_GZIP = new MediaType("application", "x-protobuf-gzip", DEFAULT_CHARSET );
public static final MediaType MEDIA_TYPE_SNAPPY = new MediaType("application", "x-protobuf-snappy",
DEFAULT_CHARSET );
 
/**
* Construct a new instance.
*/
public ProtostuffHttpMessageConverter() {
super( MEDIA_TYPE , MEDIA_TYPE_LZ4 , MEDIA_TYPE_GZIP , MEDIA_TYPE_SNAPPY );
}
 
/**
* { @inheritDoc }
*/
@Override
public boolean canRead(final Class clazz, final MediaType mediaType) {
return canRead(mediaType);
}
 
/**
* { @inheritDoc }
*/
@Override
public boolean canWrite(final Class clazz, final MediaType mediaType) {
return canWrite(mediaType);
}
 
/**
* { @inheritDoc }
*/
@Override
protected Object readInternal(final Class clazz, final HttpInputMessage inputMessage)
throws IOException, HttpMessageNotReadableException {
if ( MEDIA_TYPE .isCompatibleWith(inputMessage.getHeaders().getContentType())) {
final Schema schema = RuntimeSchema. getSchema (clazz);
 
final Object value = schema.newMessage();
 
try (final InputStream stream = inputMessage.getBody()) {
ProtobufIOUtil. mergeFrom (stream, value, (Schema) schema);
 
return value;
}
}
 
throw new HttpMessageNotReadableException(
"Unrecognized HTTP media type " + inputMessage.getHeaders().getContentType().getType() + ".");
}
 
/**
* { @inheritDoc }
*/
@Override
protected boolean supports(final Class clazz) {
// Should not be called, since we override canRead/canWrite.
throw new UnsupportedOperationException();
}
 
/**
* { @inheritDoc }
*/
@Override
protected void writeInternal(final Object o, final HttpOutputMessage outputMessage)
throws IOException, HttpMessageNotWritableException {
if(logger.isDebugEnabled()){
logger.info("Current type: {}", outputMessage.getHeaders().getContentType());
}
Stopwatch stopwatch = Stopwatch. createStarted ();
OutputStream stream = null;
 
try {
stream = outputMessage.getBody();
if ( MEDIA_TYPE .isCompatibleWith(outputMessage.getHeaders().getContentType())) {
} else if ( MEDIA_TYPE_GZIP .isCompatibleWith(outputMessage.getHeaders().getContentType())) {
stream = new GZIPOutputStream(stream);
} else if ( MEDIA_TYPE_LZ4 .isCompatibleWith(outputMessage.getHeaders().getContentType())) {
stream = new LZ4BlockOutputStream(stream);
} else if ( MEDIA_TYPE_SNAPPY .isCompatibleWith(outputMessage.getHeaders().getContentType())) {
stream = new SnappyOutputStream(stream);
} else {
throw new HttpMessageNotWritableException(
"Unrecognized HTTP media type " + outputMessage.getHeaders().getContentType().getType() + ".");
}
 
ProtobufIOUtil. writeTo (stream, o, RuntimeSchema. getSchema ((Class) o.getClass()),
LinkedBuffer. allocate ());
stream.flush();
} finally {
IOUtils. closeQuietly (stream);
}
if (logger.isDebugEnabled()){
logger.info("Output spend {}", stopwatch.toString());
}
 
}
}
springmvc配置
protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较_第3张图片
 
protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较_第4张图片
使用httpclient调用时候使用 httpBase.addHeader( "Accept" , "application/x-protobuf" ) ; //
Schema<***Dto> schema = RuntimeSchema. getSchema (***Dto.class);
***Dto value = schema.newMessage();
ProtobufIOUtil. mergeFrom (result1, value, schema);
注意的地方,
1、服务不能直接返回map或者list,如果是map或者list要经过多层封装
2、不能返回带有泛型的实体如: DataGridBean ,在服务器返回的时候没问题,
反序列化成实体时候报错,估计是
Schema<***Dto> schema = RuntimeSchema. getSchema (***Dto.class);
Schema<***Dto> schema = RuntimeSchema. getSchema (***Dto.class);//这里不能这样写,还没找到解决办法,所以只能另外写一个实体,不要使用泛型,把具体类型写进去
后来补充:对于上述问题静态已经解决了,要在声明Schema的时候使用Schema
ProtobufIOUtil第三个参数schema强转一下(Schema>)schema如下:
// DataGridBean o = new DataGridBean();
// Schema schema = RuntimeSchema.getSchema(o.getClass());
Schema schema1 = RuntimeSchema. getSchema (DataGridBean.class);
DataGridBean dataGridBean = schema1.newMessage();
ProtobufIOUtil. mergeFrom (result1, dataGridBean, (Schema>)schema1);
 
3.比较大小
只针对生产出现问题需要优化的一种常景
protostuff大小约为json的一半
protostuff
protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较_第5张图片
json
protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较_第6张图片
4.经过nginx gzip压缩后大小比较
经过fiddler抓包后 发现经过gzip压缩后大小相差不是很大
protostuff 52k左右 压缩了将近10倍
protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较_第7张图片
json 60k左右 压缩了将近20倍
protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较_第8张图片
5.在此过程中遇到到问题
5.1 使用postman调用接口的过程,postman显示的时候解压后的大小,还没找到压缩前的大小
5.2 使用httpclient4.3 返回的response会把返回 类型为gzip,x-gzip自动封装成GzipDecompressingEntity
然后remove Content-Encoding 导致使用旧的判断方法时候一直都是进入非gzip的分支
使我一度怀疑nginx的gzip不起效果
最后经过debug查看源码以及使用fildder4抓包的才确定确实有压缩
protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较_第9张图片
ResponseContentEncoding.java
protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较_第10张图片
 

你可能感兴趣的:(protobuf 、protostuff、json 经过nginx的gizp压缩前后的大小比较)