Protocol Buffer 使用
这里填写标题
- 1. Protocol Buffer 使用
-
- 1.1. Prerequisites
- 1.2. protoc 工具使用
- 1.3. protoc-gen-go
- 1.4. 参考
- 2. protobuf 特点
- 3. 使用官方版本
-
- 3.1. 安装 protobuf
- 3.2. 准备工作
- 4. 使用第三方扩展版本 gogoprotobuf
-
- 4.1. 安装插件
- 4.2. 安装 gogoprotobuf 库文件
- 4.3. 生成 go 文件
- 4.4. 性能测试
- 5. 高级应用话题
-
- 5.1. 更复杂的 Message
-
- 5.1.1. 嵌套 Message
- 5.1.2. Import Message
- 6. 测试例子: 官方例子 addressbook
-
- 6.1. 源码结构
- 6.2. 源代码
- 7. 测试例子: 个人例子, 客户端与服务端通过 HTTP 通信
-
- 7.1. 源码结构
- 7.2. 源代码
- 7.3. 运行看效果
1. Protocol Buffer 使用
1.1. Prerequisites
- Go, any one of the three latest major releases of Go.
For installation instructions, see Go’s Getting Started guide.
- Protocol buffer compiler,
protoc, version 3.
For installation instructions, see Protocol Buffer Compiler Installation.
- Go plugins for the protocol compiler:
Install the protocol compiler plugins for Go using the following commands:
$ go install google.golang.org/protobuf/cmd/[email protected]
$ go install google.golang.org/grpc/cmd/[email protected]
Update your PATH so that the protoc compiler can find the plugins:
$ export PATH="$PATH:$(go env GOPATH)/bin"
1.2. protoc 工具使用
前面我们用 protoc 来编译 .proto 文件为 go 语言, 为了支持编译为 go, 需要安装 protoc-gen-go 插件, C# 可以安装 protoc-gen-zsharp 插件。
需要注意的是, 转换 .proto 为编程语言, 不一定要安装 protoc。
例如 C# 只需要把 .proto 文件放到项目中, 通过包管理器安装一个库, 就会自动转换为相应的代码。
回归正题, 聊一下 protoc 编译 .proto 文件的命令。
protoc 常用的参数如下:
--proto_path=. #指定proto文件的路径, 填写 . 表示就在当前目录下
--go_out=. #表示编译后的文件存放路径; 如果编译的是 csharp, 则 --csharp_out
--go_opt={xxx.proto}={xxx.proto的路径} # 示例: --go_opt=Mprotos/bar.proto=example.com/project/protos/foo
最简单的编译命令:
protoc --go_out=. *.proto
--{xxx}_out 指令是必须的, 因为要输出具体的编程语言代码。
这个输出文件的路径是执行命令的路径, 如果我们不在 .proto 文件目录下执行命令, 则输出的代码便不是相同位置了。为了解决这个问题, 我们可以使用:
--go_opt=paths=source_relative
这样在别的地方执行命令, 生成的代码会跟 .proto 文件放在相同的位置。
1.3. protoc-gen-go
protoc-gen-go is a plugin for the Google protocol buffer compiler to generate Go code. Install it by building this program and making it accessible within your PATH with the name:
protoc-gen-go
The ‘go’ suffix becomes part of the argument for the protocol compiler, such that it can be invoked as:
protoc --go_out=paths=source_relative:. path/to/file.proto
This generates Go bindings for the protocol buffer defined by file.proto. With that input, the output will be written to:
path/to/file.pb.go
See the README and documentation for protocol buffers to learn more:
https://developers.google.com/protocol-buffers/
1.4. 参考
- Protocol Buffer Compiler Installation
- Quick start
2. protobuf 特点
Protocol Buffers 是一种轻便高效的结构化数据存储格式, 可以用于结构化数据串行化, 或者说序列化。它很适合做数据存储或 RPC 数据交换格式。可用于通讯协议、数据存储等领域的语言无关、平台无关、可扩展的序列化结构数据格式。
优点:
- 相比 JSON, protobuf 性能更高, 更加规范
- 编解码速度快, 数据体积小
- 使用统一的规范, 不用再担心大小写不同导致解析失败等蛋疼的问题了
缺点:
- 改动协议字段, 需要重新生成文件
- 数据没有可读性
3. 使用官方版本
3.1. 安装 protobuf
- 下载并安装 protobuf 编译器 protoc(用于编译
.proto文件)
https://github.com/protocolbuffers/protobuf/releases
- 安装编译器 protoc 插件
原生的 protoc 是不支持 golang 输出的, 所以咱们需要安装这个插件, 用于生成 .go 文件, 即使用 protoc 的命令行 --go_out=...。
$ go get github.com/golang/protobuf/protoc-gen-go
- 安装 protobuf 库文件
$ go get github.com/golang/protobuf/proto
3.2. 准备工作
- 使用下载好的编译器 protoc 编译定义好的
.proto文件
$ protoc --go_out=. *.proto
结束后会生成相应的 .pb.go 文件, 以这个文件里面的定义好的 package 名, 在 $GOPATH 下面创建一个同名文件夹, 也就是创建一个新的 golang 包了。
4. 使用第三方扩展版本 gogoprotobuf
gogoprotobuf 完全兼容 google protobuf, 它生成的代码质量和编解码性能均比 goprotobuf 高一些。
4.1. 安装插件
gogoprotobuf 两个插件可以使用:
- protoc-gen-gogo: 和 protoc-gen-go 生成的文件差不多, 性能也几乎一样 (稍微快一点点)
- protoc-gen-gofast: 生成的文件更复杂, 性能也更高 (快 5-7 倍)
//gogo
go get github.com/gogo/protobuf/protoc-gen-gogo
//gofast
go get github.com/gogo/protobuf/protoc-gen-gofast
4.2. 安装 gogoprotobuf 库文件
go get github.com/gogo/protobuf/proto
go get github.com/gogo/protobuf/gogoproto // 这个不装也没关系
4.3. 生成 go 文件
//gogo
protoc --gogo_out=. *.proto
//gofast
protoc --gofast_out=. *.proto
4.4. 性能测试
这里只是简单的用 go test 测试了一下:
//goprotobuf
"编码": 447ns/op
"解码": 422ns/op
//gogoprotobuf-go
"编码": 433ns/op
"解码": 427ns/op
//gogoprotobuf-fast
"编码": 112ns/op
"解码": 112ns/op
5. 高级应用话题
5.1. 更复杂的 Message
到这里为止, 我们只给出了一个简单的没有任何用处的例子。在实际应用中, 人们往往需要定义更加复杂的 Message。我们用"复杂"这个词, 不仅仅是指从个数上说有更多的 fields 或者更多类型的 fields, 而是指更加复杂的数据结构:
5.1.1. 嵌套 Message
嵌套是一个神奇的概念, 一旦拥有嵌套能力, 消息的表达能力就会非常强大。
代码清单 4 给出一个嵌套 Message 的例子。
message Person {
required string name = 1;
required int32 id = 2; // Unique ID number for this person.
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
在 Message Person 中, 定义了嵌套消息 PhoneNumber, 并用来定义 Person 消息中的 phone 域。这使得人们可以定义更加复杂的数据结构。
5.1.2. Import Message
在一个 .proto 文件中, 还可以用 Import 关键字引入在其他 .proto 文件中定义的消息, 这可以称做 Import Message, 或者 Dependency Message。
比如下例:
import common.header;
message youMsg{
required common.info_header header = 1;
required string youPrivateData = 2;
}
其中 ,common.info_header定义在common.header包内。
Import Message 的用处主要在于提供了方便的代码管理机制, 类似 C 语言中的头文件。您可以将一些公用的 Message 定义在一个 package 中, 然后在别的 .proto 文件中引入该 package, 进而使用其中的消息定义。
Google Protocol Buffer 可以很好地支持嵌套 Message 和引入 Message, 从而让定义复杂的数据结构的工作变得非常轻松愉快。
6. 测试例子: 官方例子 addressbook
6.1. 源码结构
$GOPATH
└── add_person
├── add_person.go
└── add_person_test.go
└── list_people
├── list_people.go
└── list_people_test.go
└── tutorial
├── addressbook.pb.go
└── addressbook.proto
6.2. 源代码
add_person/add_person.go
package main
import (
"bufio"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"strings"
pb "tutorial"
"github.com/golang/protobuf/proto"
)
func promptForAddress(r io.Reader) (*pb.Person, error) {
// A protocol buffer can be created like any struct.
p := &pb.Person{}
rd := bufio.NewReader(r)
fmt.Print("Enter person ID number:")
// An int32 field in the .proto file is represented as an int32 field
// in the generated Go struct.
if _, err := fmt.Fscanf(rd, "%d\n", &p.Id); err != nil {
return p, err
}
fmt.Print("Enter name:")
name, err := rd.ReadString('\n')
if err != nil {
return p, err
}
// A string field in the .proto file results in a string field in Go.
// We trim the whitespace because rd.ReadString includes the trailing
// newline character in its output.
p.Name = strings.TrimSpace(name)
fmt.Print("Enter email address (blank for none):")
email, err := rd.ReadString('\n')
if err != nil {
return p, err
}
p.Email = strings.TrimSpace(email)
for {
fmt.Print("Enter a phone number (or leave blank to finish):")
phone, err := rd.ReadString('\n')
if err != nil {
return p, err
}
phone = strings.TrimSpace(phone)
if phone == "" {
break
}
// The PhoneNumber message type is nested within the Person
// message in the .proto file. This results in a Go struct
// named using the name of the parent prefixed to the name of
// the nested message. Just as with pb.Person, it can be
// created like any other struct.
pn := &pb.Person_PhoneNumber{
Number: phone,
}
fmt.Print("Is this a mobile, home, or work phone?")
ptype, err := rd.ReadString('\n')
if err != nil {
return p, err
}
ptype = strings.TrimSpace(ptype)
// A proto enum results in a Go constant for each enum value.
switch ptype {
case "mobile":
pn.Type = pb.Person_MOBILE
case "home":
pn.Type = pb.Person_HOME
case "work":
pn.Type = pb.Person_WORK
default:
fmt.Printf("Unknown phone type %q. Using default.\n", ptype)
}
// A repeated proto field maps to a slice field in Go. We can
// append to it like any other slice.
p.Phones = append(p.Phones, pn)
}
return p, nil
}
// Main reads the entire address book from a file, adds one person based on
// user input, then writes it back out to the same file.
func main() {
if len(os.Args) != 2 {
log.Fatalf("Usage: %s ADDRESS_BOOK_FILE\n", os.Args[0])
}
fname := os.Args[1]
// Read the existing address book.
in, err := ioutil.ReadFile(fname)
if err != nil {
if os.IsNotExist(err) {
fmt.Printf("%s: File not found. Creating new file.\n", fname)
} else {
log.Fatalln("Error reading file:", err)
}
}
// [START marshal_proto]
book := &pb.AddressBook{}
// [START_EXCLUDE]
if err := proto.Unmarshal(in, book); err != nil {
log.Fatalln("Failed to parse address book:", err)
}
// Add an address.
addr, err := promptForAddress(os.Stdin)
if err != nil {
log.Fatalln("Error with address:", err)
}
book.People = append(book.People, addr)
// [END_EXCLUDE]
// Write the new address book back to disk.
out, err := proto.Marshal(book)
if err != nil {
log.Fatalln("Failed to encode address book:", err)
}
if err := ioutil.WriteFile(fname, out, 0644); err != nil {
log.Fatalln("Failed to write address book:", err)
}
// [END marshal_proto]
}
add_person/add_person_test.go
package main
import (
"strings"
"testing"
pb "tutorial"
"github.com/golang/protobuf/proto"
)
func TestPromptForAddressReturnsAddress(t *testing.T) {
in := `12345
Example Name
[email protected]
123-456-7890
home
222-222-2222
mobile
111-111-1111
work
777-777-7777
unknown
`
got, err := promptForAddress(strings.NewReader(in))
if err != nil {
t.Fatalf("promptForAddress(%q) had unexpected error: %s", in, err.Error())
}
if got.Id != 12345 {
t.Errorf("promptForAddress(%q) got %d, want ID %d", in, got.Id, 12345)
}
if got.Name != "Example Name" {
t.Errorf("promptForAddress(%q) => want name %q, got %q", in, "Example Name", got.Name)
}
if got.Email != "[email protected]" {
t.Errorf("promptForAddress(%q) => want email %q, got %q", in, "[email protected]", got.Email)
}
want := []*pb.Person_PhoneNumber{
{Number: "123-456-7890", Type: pb.Person_HOME},
{Number: "222-222-2222", Type: pb.Person_MOBILE},
{Number: "111-111-1111", Type: pb.Person_WORK},
{Number: "777-777-7777", Type: pb.Person_MOBILE},
}
if len(got.Phones) != len(want) {
t.Errorf("want %d phone numbers, got %d", len(want), len(got.Phones))
}
phones := len(got.Phones)
if phones > len(want) {
phones = len(want)
}
for i := 0; i < phones; i++ {
if !proto.Equal(got.Phones[i], want[i]) {
t.Errorf("want phone %q, got %q", *want[i], *got.Phones[i])
}
}
}
list_people/list_people.go
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"os"
pb "tutorial"
"github.com/golang/protobuf/proto"
)
func writePerson(w io.Writer, p *pb.Person) {
fmt.Fprintln(w, "Person ID:", p.Id)
fmt.Fprintln(w, "Name:", p.Name)
if p.Email != "" {
fmt.Fprintln(w, "E-mail address:", p.Email)
}
for _, pn := range p.Phones {
switch pn.Type {
case pb.Person_MOBILE:
fmt.Fprint(w, "Mobile phone #:")
case pb.Person_HOME:
fmt.Fprint(w, "Home phone #:")
case pb.Person_WORK:
fmt.Fprint(w, "Work phone #:")
}
fmt.Fprintln(w, pn.Number)
}
}
func listPeople(w io.Writer, book *pb.AddressBook) {
for _, p := range book.People {
writePerson(w, p)
}
}
// Main reads the entire address book from a file and prints all the
// information inside.
func main() {
if len(os.Args) != 2 {
log.Fatalf("Usage: %s ADDRESS_BOOK_FILE\n", os.Args[0])
}
fname := os.Args[1]
// [START unmarshal_proto]
// Read the existing address book.
in, err := ioutil.ReadFile(fname)
if err != nil {
log.Fatalln("Error reading file:", err)
}
book := &pb.AddressBook{}
if err := proto.Unmarshal(in, book); err != nil {
log.Fatalln("Failed to parse address book:", err)
}
// [END unmarshal_proto]
listPeople(os.Stdout, book)
}
list_people/list_people_test.go
package main
import (
"bytes"
"strings"
"testing"
pb "tutorial"
)
func TestWritePersonWritesPerson(t *testing.T) {
buf := new(bytes.Buffer)
// [START populate_proto]
p := pb.Person{
Id: 1234,
Name: "John Doe",
Email: "[email protected]",
Phones: []*pb.Person_PhoneNumber{
{Number: "555-4321", Type: pb.Person_HOME},
},
}
// [END populate_proto]
writePerson(buf, &p)
got := buf.String()
want := `Person ID: 1234
Name: John Doe
E-mail address: [email protected]
Home phone #: 555-4321
`
if got != want {
t.Errorf("writePerson(%s) =>\n\t%q, want %q", p.String(), got, want)
}
}
func TestListPeopleWritesList(t *testing.T) {
buf := new(bytes.Buffer)
in := pb.AddressBook{People: []*pb.Person{
{
Name: "John Doe",
Id: 101,
Email: "[email protected]",
},
{
Name: "Jane Doe",
Id: 102,
},
{
Name: "Jack Doe",
Id: 201,
Email: "[email protected]",
Phones: []*pb.Person_PhoneNumber{
{Number: "555-555-5555", Type: pb.Person_WORK},
},
},
{
Name: "Jack Buck",
Id: 301,
Email: "[email protected]",
Phones: []*pb.Person_PhoneNumber{
{Number: "555-555-0000", Type: pb.Person_HOME},
{Number: "555-555-0001", Type: pb.Person_MOBILE},
{Number: "555-555-0002", Type: pb.Person_WORK},
},
},
{
Name: "Janet Doe",
Id: 1001,
Email: "[email protected]",
Phones: []*pb.Person_PhoneNumber{
{Number: "555-777-0000"},
{Number: "555-777-0001", Type: pb.Person_HOME},
},
},
}}
listPeople(buf, &in)
want := strings.Split(`Person ID: 101
Name: John Doe
E-mail address: [email protected]
Person ID: 102
Name: Jane Doe
Person ID: 201
Name: Jack Doe
E-mail address: [email protected]
Work phone #: 555-555-5555
Person ID: 301
Name: Jack Buck
E-mail address: [email protected]
Home phone #: 555-555-0000
Mobile phone #: 555-555-0001
Work phone #: 555-555-0002
Person ID: 1001
Name: Janet Doe
E-mail address: [email protected]
Mobile phone #: 555-777-0000
Home phone #: 555-777-0001
`, "\n")
got := strings.Split(buf.String(), "\n")
if len(got) != len(want) {
t.Errorf(
"listPeople(%s) =>\n\t%q has %d lines, want %d",
in.String(),
buf.String(),
len(got),
len(want))
}
lines := len(got)
if lines > len(want) {
lines = len(want)
}
for i := 0; i < lines; i++ {
if got[i] != want[i] {
t.Errorf(
"listPeople(%s) =>\n\tline %d %q, want %q",
in.String(),
i,
got[i],
want[i])
}
}
}
tutorial/addressbook.proto
// See README.txt for information and build instructions.
//
// Note: START and END tags are used in comments to define sections used in
// tutorials. They are not part of the syntax for Protocol Buffers.
//
// To get an in-depth walkthrough of this file and the related examples, see:
// https://developers.google.com/protocol-buffers/docs/tutorials
// [START declaration]
syntax = "proto3";
package tutorial;
import "google/protobuf/timestamp.proto";
// [END declaration]
// [START java_declaration]
option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";
// [END java_declaration]
// [START csharp_declaration]
option csharp_namespace = "Google.Protobuf.Examples.AddressBook";
// [END csharp_declaration]
// [START messages]
message Person {
string name = 1;
int32 id = 2; // Unique ID number for this person.
string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
string number = 1;
PhoneType type = 2;
}
repeated PhoneNumber phones = 4;
google.protobuf.Timestamp last_updated = 5;
}
// Our address book file is just one of these.
message AddressBook {
repeated Person people = 1;
}
// [END messages]
7. 测试例子: 个人例子, 客户端与服务端通过 HTTP 通信
7.1. 源码结构
$GOPATH
└── proto
├── test.pb.go
└── test.proto
└── proto_client
└── client_protobuf.go
└── proto_server
└── server_protobuf.go
7.2. 源代码
proto/test.proto
syntax = "proto3"; // 指定版本, 必须要写 (proto3、proto2)
package proto;
enum FOO
{
X = 0;
};
//message 是固定的。UserInfo 是类名, 可以随意指定, 符合规范即可
message UserInfo{
string message = 1; // 消息
int32 length = 2; // 消息大小
int32 cnt = 3; // 消息计数
}
proto_client/client_protobuf.go
package main
import (
"bufio"
"fmt"
"net"
"os"
stProto "proto"
"time"
//protobuf 编解码库, 下面两个库是相互兼容的, 可以使用其中任意一个
"github.com/golang/protobuf/proto"
//"github.com/gogo/protobuf/proto"
)
func main() {
strIP := "localhost:6600"
var conn net.Conn
var err error
// 连接服务器
for conn, err = net.Dial("tcp", strIP); err != nil; conn, err = net.Dial("tcp", strIP) {
fmt.Println("connect", strIP, "fail")
time.Sleep(time.Second)
fmt.Println("reconnect...")
}
fmt.Println("connect", strIP, "success")
defer conn.Close()
// 发送消息
cnt := 0
sender := bufio.NewScanner(os.Stdin)
for sender.Scan() {
cnt++
stSend := &stProto.UserInfo{
Message: sender.Text(),
Length: *proto.Int(len(sender.Text())),
Cnt: *proto.Int(cnt),
}
//protobuf 编码
pData, err := proto.Marshal(stSend)
if err != nil {
panic(err)
}
// 发送
conn.Write(pData)
if sender.Text() == "stop" {
return
}
}
}
proto_server/server_protobuf.go
package main
import (
"fmt"
"net"
"os"
stProto "proto"
//protobuf 编解码库, 下面两个库是相互兼容的, 可以使用其中任意一个
"github.com/golang/protobuf/proto"
//"github.com/gogo/protobuf/proto"
)
func main() {
// 监听
listener, err := net.Listen("tcp", "localhost:6600")
if err != nil {
panic(err)
}
for {
conn, err := listener.Accept()
if err != nil {
panic(err)
}
fmt.Println("new connect", conn.RemoteAddr())
go readMessage(conn)
}
}
// 接收消息
func readMessage(conn net.Conn) {
defer conn.Close()
buf := make([]byte, 4096, 4096)
for {
// 读消息
cnt, err := conn.Read(buf)
if err != nil {
panic(err)
}
stReceive := &stProto.UserInfo{}
pData := buf[:cnt]
//protobuf 解码
err = proto.Unmarshal(pData, stReceive)
if err != nil {
panic(err)
}
fmt.Println("receive", conn.RemoteAddr(), stReceive)
if stReceive.Message == "stop" {
os.Exit(1)
}
}
}
7.3. 运行看效果
$ go run server_protobuf.go
$ go run client_protobuf.go
然后在客户端处输入任意字符, 在服务端可以显示出来。