protobuf 归纳
- 前言
- 一什么是protobuf
- 二简单使用示例
- 三为什么使用protobuf
- 四数据编码
- 五反射机制
- 六兼容性
- 七结束语
- 参考资料
前言
从去年新工作开始接触了protobuffer，参考当时的资料以及笔记，进行一下归纳。
一、什么是protobuf
protocol buffer( 以下简称protobuf) 是google旗下的一款平台无关，语言无关，可扩展的序列化结构数据格式。很适合用做数据存储和作为不同应用，不用语言之间相互通信的数据交换格式。
二、简单使用示例
首先，定义下面的test.proto文件
message Info{
optional string name = 1;
optional int32 age = 2;
}定义proto文件之后需要用protobuf提供的编译工具将proto文件编译成不同语言的源码，此处使用C++。
protoc -I=./ --cpp_out=./ test.proto将会生成两个文件, test.pb.h 和test.pb.cpp, 先来看一下测试代码，看如何使用protobuf，然后再来看生成的文件。
#include name()<<", age="<age()<delete pinfo;
return 0;
} 执行结果当然是
info.name=testname, age=120
下面看一下生成的test.pb.h 文件，只关注常用的部分，
// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: test.proto
#ifndef PROTOBUF_test_2eproto__INCLUDED
#define PROTOBUF_test_2eproto__INCLUDED
#include 三、为什么使用protobuf
在一些场景下，数据需要在不同的平台，不同的程序中进行传输和使用，例如某个消息是用C++程序产生的，而另一个程序是用java写的，当前者产生一个消息数据时，需要在不同的语言编写的不同的程序中进行操作，如何将消息发送并在各个程序中使用呢？这就需要设计一种消息格式，常用的就有json和xml，protobuf出现的较晚。
protobuf 的优点主要是简单，快。
protobuf将数据序列化为二进制之后，占用的空间相当小，基本仅保留了数据部分，下面章节马上提到。而xml和json会附带消息结构在数据中。
protobuf使用起来也方便，只需要反序列化就可以了，而不需要xml和json那样层层解析。
四、数据编码
上面说到了protobuf的序列化，即将结构化的数据序列化为二进制数据，方便传输或存储。
#include name()<<", age="<age()<int len = pinfo->ByteSize();
uint8_t *buf = new uint8_t[len];
pinfo->SerializeToArray(buf, len);
for(int i = 0; i < len; i++)
{
printf("%02x ",buf[i]);
}
printf("\n");
delete buf;
delete pinfo;
return 0;
} 序列化的值为：
0a 08 74 65 73 74 6e 61 6d 65 10 98 80 28在解析这些值之前，先介绍一下Varints。
varints是一种使用一个或多个字节表示整型数据的方法。其中数值本身越小，其所占用的字节数越少。
在varint中，除了最后一个字节之外的每个字节中都包含一个msb(most significant bit)设置(使用最高位)，这意味着其后的字节是否和当前字节一起来表示同一个整型数值。而字节中的其余七位将用于存储数据本身。
例如数字655384，转换为二进制为 101000 0000000 0011000，每次取七位作为数据，最高位为msb，进行Varints编码，
10011000 10000000 00101000， 16进制显示为98 80 28。（注意字节序反转，pb序列化为小端字节序）
protobuf序列化的消息是一系列key-value结构，key 采用varints编码存储，value根据不同的类型采用不同的存储方式。
| Type | Meaning | Used For | Data Len |
|---|---|---|---|
| 0 | Varint | int32, int64, uint32, uint64, sint32, sint64, bool, enum | 不固定 |
| 1 | 64-bit | fixed64, sfixed64, double | 8字节 |
| 2 | Length-delimited | string, bytes, embedded messages, packed repeated fields | key后跟随实际长度 |
| 3 | Start group | groups (deprecated) 已废弃 | |
| 4 | End group | groups (deprecated) 已废弃 | |
| 5 | 32-bit | fixed32, sfixed32, float | 4字节 |
key 值是根据字段的下标号和类型按下面的方式构成
field_number << 3 | field_type下面解析上面的输出结果：
0a 08 74 65 73 74 6e 61 6d 65 10 98 80 28
0a ===> 00001010 ===> field_num = 1, field_type = 2,
因为field_type = 2, 所以下一个值表示数据实际长度
08 ===> 数据长度为8
74['t'] 65['e'] 73['s'] 74['t'] 6e['n'] 61['a'] 6d['m'] 65['e']
10 ===> 00010000 ===> field_num = 2, field_type = 0
98 ===> 10011000 由于msb位为1，所以后面的字节表示同一个数据
98 80 28 ===> 10011000 10000000 00101000 去掉最高位，反转字节序，组合后为
010100000000000011000 ===> 655384
即：
field_num 1 = ‘testname’
field_num 2 = 655384五、反射机制
普通情况下我们获取某个字段值时都是调用对应的字段获取接口，但是有个需求需要遍历某一消息的所有字段时，该如何处理？反射机制就是提供这样的功能，但不止于此。
在之前的例子中，test.pb.h 中显示Info 类是继承自 Message类的，Message提供了两个接口，GetDescriptor/GetReflection，可以分别获取Info 的Descriptor 和Reflection。
首先看一下Descriptor 的定义，只保留了需要的代码，完整代码看protobuf-2.5.0/src/google/protobuf/descriptor.h
class LIBPROTOBUF_EXPORT Descriptor {
public:
......
// Field stuff -----------------------------------------------------
// The number of fields in this message type.
int field_count() const; //字段个数
// Gets a field by index, where 0 <= index < field_count().
// These are returned in the order they were defined in the .proto file.
const FieldDescriptor* field(int index) const; //根据索引获取字段
// Looks up a field by declared tag number. Returns NULL if no such field
// exists.
const FieldDescriptor* FindFieldByNumber(int number) const; //根据下标号获取字段
// Looks up a field by name. Returns NULL if no such field exists.
const FieldDescriptor* FindFieldByName(const string& name) const; //根据字段名获取字段
......
GOOGLE_DISALLOW_EVIL_CONSTRUCTORS(Descriptor);
};根据提供的函数，一是可以根据索引从0 - field_count() 来遍历获取消息中的每个字段； 二是，可以使用field_num 或 field_name 来获取单个字段
返回的值为FieldDescriptor, 然后再来看FieldDescriptor提供的接口
class LIBPROTOBUF_EXPORT FieldDescriptor {
public:
// Identifies a field type. 0 is reserved for errors. The order is weird
// for historical reasons. Types 12 and up are new in proto2.
//字段类型枚举
enum Type {
TYPE_DOUBLE = 1, // double, exactly eight bytes on the wire.
TYPE_FLOAT = 2, // float, exactly four bytes on the wire.
TYPE_INT64 = 3, // int64, varint on the wire. Negative numbers
// take 10 bytes. Use TYPE_SINT64 if negative
// values are likely.
TYPE_UINT64 = 4, // uint64, varint on the wire.
TYPE_INT32 = 5, // int32, varint on the wire. Negative numbers
// take 10 bytes. Use TYPE_SINT32 if negative
// values are likely.
TYPE_FIXED64 = 6, // uint64, exactly eight bytes on the wire.
TYPE_FIXED32 = 7, // uint32, exactly four bytes on the wire.
TYPE_BOOL = 8, // bool, varint on the wire.
TYPE_STRING = 9, // UTF-8 text.
TYPE_GROUP = 10, // Tag-delimited message. Deprecated.
TYPE_MESSAGE = 11, // Length-delimited message.
TYPE_BYTES = 12, // Arbitrary byte array.
TYPE_UINT32 = 13, // uint32, varint on the wire
TYPE_ENUM = 14, // Enum, varint on the wire
TYPE_SFIXED32 = 15, // int32, exactly four bytes on the wire
TYPE_SFIXED64 = 16, // int64, exactly eight bytes on the wire
TYPE_SINT32 = 17, // int32, ZigZag-encoded varint on the wire
TYPE_SINT64 = 18, // int64, ZigZag-encoded varint on the wire
MAX_TYPE = 18, // Constant useful for defining lookup tables
// indexed by Type.
};
// Specifies the C++ data type used to represent the field. There is a
// fixed mapping from Type to CppType where each Type maps to exactly one
// CppType. 0 is reserved for errors.
//字段在C++中的类型枚举
enum CppType {
CPPTYPE_INT32 = 1, // TYPE_INT32, TYPE_SINT32, TYPE_SFIXED32
CPPTYPE_INT64 = 2, // TYPE_INT64, TYPE_SINT64, TYPE_SFIXED64
CPPTYPE_UINT32 = 3, // TYPE_UINT32, TYPE_FIXED32
CPPTYPE_UINT64 = 4, // TYPE_UINT64, TYPE_FIXED64
CPPTYPE_DOUBLE = 5, // TYPE_DOUBLE
CPPTYPE_FLOAT = 6, // TYPE_FLOAT
CPPTYPE_BOOL = 7, // TYPE_BOOL
CPPTYPE_ENUM = 8, // TYPE_ENUM
CPPTYPE_STRING = 9, // TYPE_STRING, TYPE_BYTES
CPPTYPE_MESSAGE = 10, // TYPE_MESSAGE, TYPE_GROUP
MAX_CPPTYPE = 10, // Constant useful for defining lookup tables
// indexed by CppType.
};
// Identifies whether the field is optional, required, or repeated. 0 is
// reserved for errors.
//字段属性枚举
enum Label {
LABEL_OPTIONAL = 1, // optional
LABEL_REQUIRED = 2, // required
LABEL_REPEATED = 3, // repeated
MAX_LABEL = 3, // Constant useful for defining lookup tables
// indexed by Label.
};
// Valid field numbers are positive integers up to kMaxNumber.
//字段所支持的最大下标号，一个消息中的字段的下标不能大于此值。
static const int kMaxNumber = (1 << 29) - 1;
// 获取字段名
const string& name() const; // Name of this field within the message.
const string& full_name() const; // Fully-qualified name of the field.
// 获取下标号
int number() const; // Declared tag number.
// Same as name() except converted to lower-case. This (and especially the
// FindFieldByLowercaseName() method) can be useful when parsing formats
// which prefer to use lowercase naming style. (Although, technically
// field names should be lowercased anyway according to the protobuf style
// guide, so this only makes a difference when dealing with old .proto files
// which do not follow the guide.)
// 获取字段全小写名
const string& lowercase_name() const;
// 字段的类型枚举值
Type type() const; // Declared type of this field.
// 字段类型名
const char* type_name() const; // Name of the declared type.
// 字段cpp类型枚举值
CppType cpp_type() const; // C++ type of this field.
// 字段cpp类型名
const char* cpp_type_name() const; // Name of the C++ type.
// 字段属性
Label label() const; // optional/required/repeated
bool is_required() const; // shorthand for label() == LABEL_REQUIRED
bool is_optional() const; // shorthand for label() == LABEL_OPTIONAL
bool is_repeated() const; // shorthand for label() == LABEL_REPEATED
bool is_packable() const; // shorthand for is_repeated() &&
// IsTypePackable(type())
bool is_packed() const; // shorthand for is_packable() &&
// options().packed()
// Index of this field within the message's field array, or the file or
// extension scope's extensions array.
//字段的索引值
int index() const;
// Does this field have an explicitly-declared default value?
// 是否有默认值
bool has_default_value() const;
// 获取默认值接口
int32 default_value_int32() const;
int64 default_value_int64() const;
uint32 default_value_uint32() const;
uint64 default_value_uint64() const;
float default_value_float() const;
double default_value_double() const;
bool default_value_bool() const;
const EnumValueDescriptor* default_value_enum() const;
const string& default_value_string() const;
// The Descriptor for the message of which this is a field. For extensions,
// this is the extended type. Never NULL.
// 获取该字段位于的消息
const Descriptor* containing_type() const;
......
GOOGLE_DISALLOW_EVIL_CONSTRUCTORS(FieldDescriptor);
};可以看到字段的一些属性在这里都能看到(字段类型，字段索引，字段下标，字段名，字段属性)
接下来的Reflection 类就将提供如何取字段值了，只保留了常用接口，完整代码请查看protobuf-2.5.0/src/google/protobuf/message.h
class LIBPROTOBUF_EXPORT Reflection {
public:
......
// Estimate the amount of memory used by the message object.
// 数据大小
virtual int SpaceUsed(const Message& message) const = 0;
// Check if the given non-repeated field is set.
// 是否包含field 字段
virtual bool HasField(const Message& message,
const FieldDescriptor* field) const = 0;
// Get the number of elements of a repeated field.
// field字段的大小
virtual int FieldSize(const Message& message,
const FieldDescriptor* field) const = 0;
// Clear the value of a field, so that HasField() returns false or
// FieldSize() returns zero.
// 清除field的值
virtual void ClearField(Message* message,
const FieldDescriptor* field) const = 0;
......
// Singular field getters ------------------------------------------
// These get the value of a non-repeated field. They return the default
// value for fields that aren't set.
// 获取字段的值，根据field的type调用不同接口
virtual int32 GetInt32 (const Message& message,
const FieldDescriptor* field) const = 0;
virtual int64 GetInt64 (const Message& message,
const FieldDescriptor* field) const = 0;
virtual uint32 GetUInt32(const Message& message,
const FieldDescriptor* field) const = 0;
virtual uint64 GetUInt64(const Message& message,
const FieldDescriptor* field) const = 0;
virtual float GetFloat (const Message& message,
const FieldDescriptor* field) const = 0;
virtual double GetDouble(const Message& message,
const FieldDescriptor* field) const = 0;
virtual bool GetBool (const Message& message,
const FieldDescriptor* field) const = 0;
virtual string GetString(const Message& message,
const FieldDescriptor* field) const = 0;
virtual const EnumValueDescriptor* GetEnum(
const Message& message, const FieldDescriptor* field) const = 0;
// See MutableMessage() for the meaning of the "factory" parameter.
virtual const Message& GetMessage(const Message& message,
const FieldDescriptor* field,
MessageFactory* factory = NULL) const = 0;
// Get a string value without copying, if possible.
//
// GetString() necessarily returns a copy of the string. This can be
// inefficient when the string is already stored in a string object in the
// underlying message. GetStringReference() will return a reference to the
// underlying string in this case. Otherwise, it will copy the string into
// *scratch and return that.
//
// Note: It is perfectly reasonable and useful to write code like:
// str = reflection->GetStringReference(field, &str);
// This line would ensure that only one copy of the string is made
// regardless of the field's underlying representation. When initializing
// a newly-constructed string, though, it's just as fast and more readable
// to use code like:
// string str = reflection->GetString(field);
virtual const string& GetStringReference(const Message& message,
const FieldDescriptor* field,
string* scratch) const = 0;
// Singular field mutators -----------------------------------------
// These mutate the value of a non-repeated field.
// 设置字段的值，
virtual void SetInt32 (Message* message,
const FieldDescriptor* field, int32 value) const = 0;
virtual void SetInt64 (Message* message,
const FieldDescriptor* field, int64 value) const = 0;
virtual void SetUInt32(Message* message,
const FieldDescriptor* field, uint32 value) const = 0;
virtual void SetUInt64(Message* message,
const FieldDescriptor* field, uint64 value) const = 0;
virtual void SetFloat (Message* message,
const FieldDescriptor* field, float value) const = 0;
virtual void SetDouble(Message* message,
const FieldDescriptor* field, double value) const = 0;
virtual void SetBool (Message* message,
const FieldDescriptor* field, bool value) const = 0;
virtual void SetString(Message* message,
const FieldDescriptor* field,
const string& value) const = 0;
virtual void SetEnum (Message* message,
const FieldDescriptor* field,
const EnumValueDescriptor* value) const = 0;
// Get a mutable pointer to a field with a message type. If a MessageFactory
// is provided, it will be used to construct instances of the sub-message;
// otherwise, the default factory is used. If the field is an extension that
// does not live in the same pool as the containing message's descriptor (e.g.
// it lives in an overlay pool), then a MessageFactory must be provided.
// If you have no idea what that meant, then you probably don't need to worry
// about it (don't provide a MessageFactory). WARNING: If the
// FieldDescriptor is for a compiled-in extension, then
// factory->GetPrototype(field->message_type() MUST return an instance of the
// compiled-in class for this type, NOT DynamicMessage.
virtual Message* MutableMessage(Message* message,
const FieldDescriptor* field,
MessageFactory* factory = NULL) const = 0;
// Releases the message specified by 'field' and returns the pointer,
// ReleaseMessage() will return the message the message object if it exists.
// Otherwise, it may or may not return NULL. In any case, if the return value
// is non-NULL, the caller takes ownership of the pointer.
// If the field existed (HasField() is true), then the returned pointer will
// be the same as the pointer returned by MutableMessage().
// This function has the same effect as ClearField().
virtual Message* ReleaseMessage(Message* message,
const FieldDescriptor* field,
MessageFactory* factory = NULL) const = 0;
// Repeated field getters ------------------------------------------
// These get the value of one element of a repeated field.
virtual int32 GetRepeatedInt32 (const Message& message,
const FieldDescriptor* field,
int index) const = 0;
virtual int64 GetRepeatedInt64 (const Message& message,
const FieldDescriptor* field,
int index) const = 0;
virtual uint32 GetRepeatedUInt32(const Message& message,
const FieldDescriptor* field,
int index) const = 0;
virtual uint64 GetRepeatedUInt64(const Message& message,
const FieldDescriptor* field,
int index) const = 0;
virtual float GetRepeatedFloat (const Message& message,
const FieldDescriptor* field,
int index) const = 0;
virtual double GetRepeatedDouble(const Message& message,
const FieldDescriptor* field,
int index) const = 0;
virtual bool GetRepeatedBool (const Message& message,
const FieldDescriptor* field,
int index) const = 0;
virtual string GetRepeatedString(const Message& message,
const FieldDescriptor* field,
int index) const = 0;
virtual const EnumValueDescriptor* GetRepeatedEnum(
const Message& message,
const FieldDescriptor* field, int index) const = 0;
virtual const Message& GetRepeatedMessage(
const Message& message,
const FieldDescriptor* field, int index) const = 0;
// See GetStringReference(), above.
virtual const string& GetRepeatedStringReference(
const Message& message, const FieldDescriptor* field,
int index, string* scratch) const = 0;
// Repeated field mutators -----------------------------------------
// These mutate the value of one element of a repeated field.
virtual void SetRepeatedInt32 (Message* message,
const FieldDescriptor* field,
int index, int32 value) const = 0;
virtual void SetRepeatedInt64 (Message* message,
const FieldDescriptor* field,
int index, int64 value) const = 0;
virtual void SetRepeatedUInt32(Message* message,
const FieldDescriptor* field,
int index, uint32 value) const = 0;
virtual void SetRepeatedUInt64(Message* message,
const FieldDescriptor* field,
int index, uint64 value) const = 0;
virtual void SetRepeatedFloat (Message* message,
const FieldDescriptor* field,
int index, float value) const = 0;
virtual void SetRepeatedDouble(Message* message,
const FieldDescriptor* field,
int index, double value) const = 0;
virtual void SetRepeatedBool (Message* message,
const FieldDescriptor* field,
int index, bool value) const = 0;
virtual void SetRepeatedString(Message* message,
const FieldDescriptor* field,
int index, const string& value) const = 0;
virtual void SetRepeatedEnum(Message* message,
const FieldDescriptor* field, int index,
const EnumValueDescriptor* value) const = 0;
// Get a mutable pointer to an element of a repeated field with a message
// type.
virtual Message* MutableRepeatedMessage(
Message* message, const FieldDescriptor* field, int index) const = 0;
// Repeated field adders -------------------------------------------
// These add an element to a repeated field.
virtual void AddInt32 (Message* message,
const FieldDescriptor* field, int32 value) const = 0;
virtual void AddInt64 (Message* message,
const FieldDescriptor* field, int64 value) const = 0;
virtual void AddUInt32(Message* message,
const FieldDescriptor* field, uint32 value) const = 0;
virtual void AddUInt64(Message* message,
const FieldDescriptor* field, uint64 value) const = 0;
virtual void AddFloat (Message* message,
const FieldDescriptor* field, float value) const = 0;
virtual void AddDouble(Message* message,
const FieldDescriptor* field, double value) const = 0;
virtual void AddBool (Message* message,
const FieldDescriptor* field, bool value) const = 0;
virtual void AddString(Message* message,
const FieldDescriptor* field,
const string& value) const = 0;
virtual void AddEnum (Message* message,
const FieldDescriptor* field,
const EnumValueDescriptor* value) const = 0;
// See MutableMessage() for comments on the "factory" parameter.
virtual Message* AddMessage(Message* message,
const FieldDescriptor* field,
MessageFactory* factory = NULL) const = 0;
......
GOOGLE_DISALLOW_EVIL_CONSTRUCTORS(Reflection);
};基于以上内容，下面修改test.cpp 使用反射机制遍历所有字段进行取值。
#include " "<delete pinfo;
return 0;
} 运行结果：
field name = testname
field age = 655384六、兼容性
在实际的开发中会存在这样一种应用场景，既消息格式因为某些需求的变化而不得不进行必要的升级，但是有些使用原有消息格式的应用程序暂时又不能被立刻升级，这便要求我们在升级消息格式时要遵守一定的规则，从而可以保证基于新老消息格式的新老程序同时运行。规则如下：
1. 不要修改已经存在字段的标签号。
2. 任何新添加的字段必须是optional和repeated限定符，否则无法保证新老程序在互相传递消息时的消息兼容性。
3. 在原有的消息中，不能移除已经存在的required字段，optional和repeated类型的字段可以被移除，但是他们之前使用的标签号必须被保留，不能被新的字段重用。
4. int32、uint32、int64、uint64和bool等类型之间是兼容的，sint32和sint64是兼容的，string和bytes是兼容的，fixed32和sfixed32，以及fixed64和sfixed64之间是兼容的，这意味着如果想修改原有字段的类型时，为了保证兼容性，只能将其修改为与其原有类型兼容的类型，否则就将打破新老消息格式的兼容性。
5. optional和repeated限定符也是相互兼容的。
七、结束语
本文没有介绍protobuf的语法，想了解的同学自己百度吧。
参考资料：
1.基础介绍
Protocol Buffer 的使用和原理
Protocol Buffer技术详解(语言规范)
Protocol Buffer技术详解(C++实例)
2.反射机制
一种自动反射消息类型的 Google Protobuf 网络传输方案[陈硕]
Protobuf反射详解
Google Protocol Buffer 的使用和原理
反射机制学习笔记
3.数据编码
Protocol Buffer技术详解(数据编码)