gRPC之grpc keepalive

1、grpc keepalive

keepalive ping是一种通过transport发送HTTP2 ping来检查通道当前是否工作的方法。它是周期性发送的，

如果在某个超时周期内该ping没有得到对等方的确认，则传输断开连接。

gRPC keepAlive是grpc框架在应用层面连接保活的一种措施，即当grpc连接上没有业务数据时，是否发送ping

pong，以保持连接活跃性，不因长时间空闲而被Server或操作系统关闭。

gRPC keepAlive在client与server都有，client端默认关闭(keepAliveTime为Long.MAX_VALUE)，server端默认打

开，keepAliveTime为2小时，即每2小时向client发送一次ping。

客户端和服务端都可以发送ping帧，接收端则回复带ACK flag的ping帧。

Timeout：ping帧的发送端发送ping帧之后，会等待一段时间，如果在这段时间里没有收到对端的回复(带有ack

标志的ping帧)，则认为连接已经关闭。

有关如何配置keepalive，请参考：

https://pkg.go.dev/google.golang.org/grpc/keepalive

https://pkg.go.dev/google.golang.org/grpc?utm_source=godoc#WithKeepaliveParams

1.1 keepalive参数说明

1.1.1 客户端配置

对于客户端来说，在拨号之前，使用下面的数据结构配置 keepalive参数：

type ClientParameters struct {
   // After a duration of this time if the client doesn't see any activity it
   // pings the server to see if the transport is still alive.
   // If set below 10s, a minimum value of 10s will be used instead.
   Time time.Duration // The current default value is infinity.
   // After having pinged for keepalive check, the client waits for a duration
   // of Timeout and if no activity is seen even after that the connection is
   // closed.
   Timeout time.Duration // The current default value is 20 seconds.
   // If true, client sends keepalive pings even with no active RPCs. If false,
   // when there are no active RPCs, Time and Timeout will be ignored and no
   // keepalive pings will be sent.
   PermitWithoutStream bool // false by default.
}

解释：

Time：超过这个时长都没有活动的话，客户端就会ping服务端，这个值最小是10秒。

Timeout ：发出Ping后，客户端等待回复，如果超过这个时长没有收到ping的回复消息，则会断开链接，默

认值是20秒。

PermitWithoutStream：即使没有活动流也发送ping。

Time是客户端发送ping帧之前，连接空闲的时间。PermitWithoutStream 这个值规定了当连接上没有RPC调用时

是否可以发送ping帧。

可以通过函数WithKeepaliveParams 设置：

var kacp = keepalive.ClientParameters{
	Time: 10 * time.Second,
	Timeout: time.Second,
	PermitWithoutStream: true,
}

conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithKeepaliveParams(kacp))

1.1.2 服务端keepalive参数配置

// ServerParameters is used to set keepalive and max-age parameters on the
// server-side.
type ServerParameters struct {
	// MaxConnectionIdle is a duration for the amount of time after which an
	// idle connection would be closed by sending a GoAway. Idleness duration is
	// defined since the most recent time the number of outstanding RPCs became
	// zero or the connection establishment.
	MaxConnectionIdle time.Duration // The current default value is infinity.
	// MaxConnectionAge is a duration for the maximum amount of time a
	// connection may exist before it will be closed by sending a GoAway. A
	// random jitter of +/-10% will be added to MaxConnectionAge to spread out
	// connection storms.
	MaxConnectionAge time.Duration // The current default value is infinity.
	// MaxConnectionAgeGrace is an additive period after MaxConnectionAge after
	// which the connection will be forcibly closed.
	MaxConnectionAgeGrace time.Duration // The current default value is infinity.
	// After a duration of this time if the server doesn't see any activity it
	// pings the client to see if the transport is still alive.
	// If set below 1s, a minimum value of 1s will be used instead.
	Time time.Duration // The current default value is 2 hours.
	// After having pinged for keepalive check, the server waits for a duration
	// of Timeout and if no activity is seen even after that the connection is
	// closed.
	Timeout time.Duration // The current default value is 20 seconds.
}

解释：

MaxConnectionIdle：当连接处于idle的时长超过MaxConnectionIdle时，服务端就发送GOAWAY，关闭连

接，该值的默认值为无限大。

MaxConnectionAge：一个连接只能使用MaxConnectionAge这么长的时间，否则服务端就会关闭这个连

接。

MaxConnectionAgeGrace：服务端优雅关闭连接时长。

Time：超过这个时长都没有活动的话，服务端就会ping客户端，默认值为2小时。

Timeout：服务端发送ping请求后，等待客户端响应的时间，若无响应则将该链接关闭回收，默认值为20

秒。

服务端配置的 Time 和 Timeout的含义和客户端配置相同。除此之外，要有3个配置可以影响一个连接：

MaxConnectionIdle：连接的最大空闲时长。当超过这个时间时，服务端会向客户端发送GOAWAY帧，关闭空

闲的连接，节省连接数。

MaxConnectionAge：一个连接可以使用的时间。当一个连接已经使用了超过这个值的时间时，服务端就要强制

关闭连接了。如果客户端仍然要连接服务端，可以重新发起连接。这时连接将进入半关闭状态，不再接收新的流。

MaxConnectionAgeGrace当服务端决定关闭一个连接时，如果有RPC在进行，会等待MaxConnectionAgeGrace

时间，让已经存在的流可以正常处理完毕。

为了保护服务端，防止恶意攻击或者防止客户端不去恰当的行为，对服务端造成破坏或性能受影响，服务端还针对

keepalive设计了一个策略，叫 EnforcementPolicy，可以限制客户端ping的频率。

1.1.3 服务端EnforcementPolicy配置

EnforcementPolicy的配置，用于在服务器端设置 keepalive 强制策略。服务器将关闭与违反此策略的客户端的

连接。

// EnforcementPolicy is used to set keepalive enforcement policy on the
// server-side. Server will close connection with a client that violates this
// policy.
type EnforcementPolicy struct {
	// MinTime is the minimum amount of time a client should wait before sending
	// a keepalive ping.
	MinTime time.Duration // The current default value is 5 minutes.
	// If true, server allows keepalive pings even when there are no active
	// streams(RPCs). If false, and client sends ping when there are no active
	// streams, server will send GOAWAY and close the connection.
	PermitWithoutStream bool // false by default.
}

解释：

MinTime：客户端ping的间隔应该不小于这个时长，默认是5分钟。

PermitWithoutStream：服务端是否允许在没有RPC调用时发送PING，默认不允许。在不允许的情况下，

客户端发送了PING，服务端将发送GOAWAY帧，关闭连接。

如果客户端在 MinTime 时间内发送了1次以上的ping，或者在服务端PermitWithoutStream为 false且连接上没

有RPC进行时，服务端收到ping帧，则会关闭连接。

1.1.4 服务端配置

服务端配置有这两个参数：

type serverOptions struct {
    keepaliveParams       keepalive.ServerParameters
    keepalivePolicy       keepalive.EnforcementPolicy
}

在启动server之前，可以通过 KeepaliveParams 和 KeepaliveEnforcementPolicy 这两个函数配置。

var kaep = keepalive.EnforcementPolicy{
	MinTime: 5 * time.Second,
	PermitWithoutStream: true,
}

var kasp = keepalive.ServerParameters{
	MaxConnectionIdle: 15 * time.Second,
	MaxConnectionAge: 30 * time.Second,
	MaxConnectionAgeGrace: 5 * time.Second,
	Time: 5 * time.Second,
	Timeout: 1 * time.Second,
}

s := grpc.NewServer(grpc.KeepaliveEnforcementPolicy(kaep), grpc.KeepaliveParams(kasp))

下面通过示例说明如何设置客户端保活ping和服务器端保活ping强制和连接空闲设置。

1.2 keepalive实例1

1.2.1 proto编写和编译

syntax = "proto3";

option go_package = "./;echo";

package echo;

message EchoRequest {
    string message = 1;
}

message EchoResponse {
    string message = 1;
}

service Echo {
    rpc UnaryEcho(EchoRequest) returns (EchoResponse) {}
}

$ protoc -I . --go_out=plugins=grpc:. ./echo.proto

1.2.2 服务端

package main

import (
	"context"
	pb "demo/pb"
	"flag"
	"fmt"
	"google.golang.org/grpc"
	"google.golang.org/grpc/keepalive"
	"log"
	"net"
	"time"
)

var port = flag.Int("port", 50052, "port number")

var kaep = keepalive.EnforcementPolicy{
	// If a client pings more than once every 5 seconds, terminate the connection
	MinTime: 5 * time.Second,
	// Allow pings even when there are no active streams
	PermitWithoutStream: true,
}

var kasp = keepalive.ServerParameters{
	// If a client is idle for 15 seconds, send a GOAWAY
	MaxConnectionIdle: 15 * time.Second,
	// If any connection is alive for more than 30 seconds, send a GOAWAY
	MaxConnectionAge: 30 * time.Second,
	// Allow 5 seconds for pending RPCs to complete before forcibly closing connections
	MaxConnectionAgeGrace: 5 * time.Second,
	// Ping the client if it is idle for 5 seconds to ensure the connection is still active
	Time: 5 * time.Second,
	// Wait 1 second for the ping ack before assuming the connection is dead
	Timeout: 1 * time.Second,
}

// server implements EchoServer.
type server struct {
	pb.UnimplementedEchoServer
}

func (s *server) UnaryEcho(ctx context.Context, req *pb.EchoRequest) (*pb.EchoResponse, error) {
	return &pb.EchoResponse{Message: req.Message}, nil
}

func main() {
	flag.Parse()
	address := fmt.Sprintf(":%v", *port)
	lis, err := net.Listen("tcp", address)
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	s := grpc.NewServer(grpc.KeepaliveEnforcementPolicy(kaep), grpc.KeepaliveParams(kasp))
	pb.RegisterEchoServer(s, &server{})
	if err := s.Serve(lis); err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}

1.2.3 客户端

package main

import (
	"context"
	pb "demo/pb"
	"flag"
	"fmt"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/keepalive"
	"log"
	"time"
)

var addr = flag.String("addr", "localhost:50052", "the address to connect to")

var kacp = keepalive.ClientParameters{
	// send pings every 10 seconds if there is no activity
	Time: 10 * time.Second,
	// wait 1 second for ping ack before considering the connection dead
	Timeout: time.Second,
	// send pings even without active streams
	PermitWithoutStream: true,
}

func main() {
	flag.Parse()
	conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithKeepaliveParams(kacp))
	if err != nil {
		log.Fatalf("did not connect: %v", err)
	}
	defer conn.Close()
	c := pb.NewEchoClient(conn)
	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
	defer cancel()
	fmt.Println("Performing unary request")
	res, err := c.UnaryEcho(ctx, &pb.EchoRequest{Message: "keepalive demo"})
	if err != nil {
		log.Fatalf("unexpected error from UnaryEcho: %v", err)
	}
	fmt.Println("RPC response:", res)
	select {}
	// Block forever; run with GODEBUG=http2debug=2 to observe ping frames and GOAWAYs due to idleness.
}

1.2.4 测试

[root@zsx demo]# go run server/server.go

[root@zsx demo]# env GODEBUG=http2debug=2 go run client/client.go
Performing unary request
2023/02/18 10:24:18 http2: Framer 0xc000166000: wrote SETTINGS len=0
2023/02/18 10:24:18 http2: Framer 0xc000166000: read SETTINGS len=6, settings: MAX_FRAME_SIZE=16384
2023/02/18 10:24:18 http2: Framer 0xc000166000: read SETTINGS flags=ACK len=0
2023/02/18 10:24:18 http2: Framer 0xc000166000: wrote SETTINGS flags=ACK len=0
2023/02/18 10:24:18 http2: Framer 0xc000166000: wrote HEADERS flags=END_HEADERS stream=1 len=86
2023/02/18 10:24:18 http2: Framer 0xc000166000: wrote DATA flags=END_STREAM stream=1 len=21 data="\x00\x00\x00\x00\x10\n\x0ekeepalive demo"
2023/02/18 10:24:18 http2: Framer 0xc000166000: read WINDOW_UPDATE len=4 (conn) incr=21
2023/02/18 10:24:18 http2: Framer 0xc000166000: read PING len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2023/02/18 10:24:18 http2: Framer 0xc000166000: read HEADERS flags=END_HEADERS stream=1 len=14
2023/02/18 10:24:18 http2: decoded hpack field header field ":status" = "200"
2023/02/18 10:24:18 http2: decoded hpack field header field "content-type" = "application/grpc"
2023/02/18 10:24:18 http2: Framer 0xc000166000: read DATA stream=1 len=21 data="\x00\x00\x00\x00\x10\n\x0ekeepalive demo"
2023/02/18 10:24:18 http2: Framer 0xc000166000: read HEADERS flags=END_STREAM|END_HEADERS stream=1 len=24
2023/02/18 10:24:18 http2: decoded hpack field header field "grpc-status" = "0"
2023/02/18 10:24:18 http2: decoded hpack field header field "grpc-message" = ""
RPC response: message:"keepalive demo"
2023/02/18 10:24:18 http2: Framer 0xc000166000: wrote PING flags=ACK len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2023/02/18 10:24:18 http2: Framer 0xc000166000: wrote WINDOW_UPDATE len=4 (conn) incr=21
2023/02/18 10:24:18 http2: Framer 0xc000166000: wrote PING len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2023/02/18 10:24:18 http2: Framer 0xc000166000: read PING flags=ACK len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2023/02/18 10:24:23 http2: Framer 0xc000166000: read PING len=8 ping="\x00\x00\x00\x00\x00\x00\x00\x00"
2023/02/18 10:24:23 http2: Framer 0xc000166000: wrote PING flags=ACK len=8 ping="\x00\x00\x00\x00\x00\x00\x00\x00"
2023/02/18 10:24:28 http2: Framer 0xc000166000: read PING len=8 ping="\x00\x00\x00\x00\x00\x00\x00\x00"
2023/02/18 10:24:28 http2: Framer 0xc000166000: wrote PING flags=ACK len=8 ping="\x00\x00\x00\x00\x00\x00\x00\x00"
2023/02/18 10:24:33 http2: Framer 0xc000166000: read GOAWAY len=8 LastStreamID=2147483647 ErrCode=NO_ERROR Debug=""
2023/02/18 10:24:33 http2: Framer 0xc000166000: read PING len=8 ping="\x01\x06\x01\b\x00\x03\x03\t"

第一个PING帧是服务端发起的。

因为服务端每当连接空闲5秒就发送ping帧，客户端配置为10秒。服务端在发送第一个PING之后5秒，就发送了第

二个ping帧。

当时间到10:24:33的时候，服务端检测到此连接已经持续空闲15秒了，达到 MaxConnectionIdle的值了，而且

此时没有进行中的RPC，因此发送GOAWAY帧，关闭连接。

修改客户端和服务端的配置，就很容易看到客户端和服务端都在向对方发送PING帧的过程。

# 项目结构
$ tree demo/
demo/
├── client
│   └── client.go
├── go.mod
├── go.sum
├── pb
│   ├── echo.pb.go
│   └── echo.proto
└── server
    └── server.go

3 directories, 6 files

1.3 keepalive实例2

1.3.1 proto编写和编译

syntax = "proto3";
package pb;
option go_package = "./;pb";

service Greeter {
    rpc SayHello (HelloRequest) returns (HelloReply) {}
}

message HelloRequest {
    string name = 1;
}

message HelloReply {
    string message = 1;
}

$ protoc -I . --go_out=plugins=grpc:. ./helloword.proto

1.3.2 服务端

package main

import (
	"context"
	pb "demo/pb"
	"google.golang.org/grpc"
	"google.golang.org/grpc/keepalive"
	"log"
	"net"
	"time"
)

const (
	port = ":50051"
)

type server struct {
	pb.UnimplementedGreeterServer
}

var kaep = keepalive.EnforcementPolicy{
	// If a client pings more than once every 5 seconds, terminate the connection
	MinTime: 5 * time.Second,
	// Allow pings even when there are no active streams
	PermitWithoutStream: true,
}

// 该函数定义必须与helloworld.pb.go 定义的SayHello一致
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
	//打印客户端传入HelloRequest请求的Name参数
	log.Printf("Received: %v", in.GetName())
	time.Sleep(time.Hour)
	//将name参数作为返回值，返回给客户端
	return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}

// main方法 函数开始执行的地方
func main() {
	// 调用标准库，监听50051端口的tcp连接
	lis, err := net.Listen("tcp", port)
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	//创建grpc服务
	s := grpc.NewServer(grpc.KeepaliveEnforcementPolicy(kaep))
	//将server对象，也就是实现SayHello方法的对象，与grpc服务绑定
	pb.RegisterGreeterServer(s, &server{})
	// grpc服务开始接收访问50051端口的tcp连接数据
	if err := s.Serve(lis); err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}

1.3.3 客户端

package main

import (
	"context"
	"demo/pb"
	"fmt"
	"google.golang.org/grpc"
	"google.golang.org/grpc/keepalive"
	"google.golang.org/grpc/stats"
	"log"
	"time"
)

var kacp = keepalive.ClientParameters{
	// send pings every 10 seconds if there is no activity
	Time: 15 * time.Second,
	// wait 1 second for ping ack before considering the connection dead
	Timeout: time.Second,
	// send pings even without active streams
	PermitWithoutStream: true,
}

const (
	address = "localhost:50051"
)

func main() {
	// 访问服务端address，创建连接conn
	conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithBlock(), grpc.WithKeepaliveParams(kacp),
		grpc.WithStatsHandler(&StatsHandler{}))
	if err != nil {
		log.Fatalf("did not connect: %v", err)
	}
	defer conn.Close()
	c := pb.NewGreeterClient(conn)
	// 设置客户端访问超时时间1秒
	ctx, cancel := context.WithTimeout(context.Background(), 100*time.Second)
	defer cancel()
	// 客户端调用服务端 SayHello 请求，传入Name 为 "world", 返回值为服务端返回参数
	r, err := c.SayHello(ctx, &pb.HelloRequest{Name: "world"})
	if err != nil {
		log.Fatalf("could not greet: %v", err)
	}
	// 根据服务端处理逻辑，返回值也为"world"
	log.Printf("Greeting: %s", r.GetMessage())
}

type StatsHandler struct {
}

// TagConn可以将一些信息附加到给定的上下文。
func (h *StatsHandler) TagConn(ctx context.Context, info *stats.ConnTagInfo) context.Context {
	fmt.Printf("TagConn:%v\n", info)
	return ctx
}

// 会在连接开始和结束时被调用，分别会输入不同的状态.
func (h *StatsHandler) HandleConn(ctx context.Context, s stats.ConnStats) {
	fmt.Printf("HandleConn:%v\n", s)
}

// TagRPC可以将一些信息附加到给定的上下文
func (h *StatsHandler) TagRPC(ctx context.Context, info *stats.RPCTagInfo) context.Context {
	fmt.Printf("TagRPC:%v\n", info)
	return ctx
}

// 处理RPC统计信息
func (h *StatsHandler) HandleRPC(ctx context.Context, s stats.RPCStats) {
	fmt.Printf("HandleRPC:%v\n", s)
}

1.3.4 测试

[root@zsx demo]# go run server/server.go
2023/02/18 12:31:13 Received: world

[root@zsx demo]# go run client/client.go
TagConn:&{[::1]:50051 [::1]:63094}
HandleConn:&{true}
TagRPC:&{/pb.Greeter/SayHello true}
HandleRPC:&{true 2023-02-18 12:31:13.9589101 +0800 CST m=+0.018497101 true false false false}
HandleRPC:&{true  map[user-agent:[grpc-go/1.53.0]] /pb.Greeter/SayHello [::1]:50051 [::1]:63094}
HandleRPC:&{true name:"world" [10 5 119 111 114 108 100] 7 12 2023-02-18 12:31:13.9594239 +0800 CST m=+0.019010901}
HandleRPC:&{true 2023-02-18 12:31:13.9589101 +0800 CST m=+0.018497101 2023-02-18 12:32:53.9592626 +0800 CST m=+100.018849601 map[] rpc
 error: code = DeadlineExceeded desc = context deadline exceeded}
2023/02/18 12:32:53 could not greet: rpc error: code = DeadlineExceeded desc = context deadline exceeded
exit status 1

如果停止服务端，客户端还会打印：

HandleConn:&{true}

如果去掉服务端的 time.Sleep(time.Hour)：

[root@zsx demo]# go run server/server.go
2023/02/18 12:37:40 Received: world

[root@zsx demo]# go run client/client.go
TagConn:&{[::1]:50051 [::1]:63365}
HandleConn:&{true}
TagRPC:&{/pb.Greeter/SayHello true}
HandleRPC:&{true 2023-02-18 12:37:40.2493083 +0800 CST m=+0.017940701 true false false false}
HandleRPC:&{true  map[user-agent:[grpc-go/1.53.0]] /pb.Greeter/SayHello [::1]:50051 [::1]:63365}
HandleRPC:&{true name:"world" [10 5 119 111 114 108 100] 7 12 2023-02-18 12:37:40.2498263 +0800 CST m=+0.018458701}
HandleRPC:&{true 14  map[content-type:[application/grpc]]  <nil> <nil>}
HandleRPC:&{true 24 map[]}
HandleRPC:&{true message:"Hello world" [10 11 72 101 108 108 111 32 119 111 114 108 100] 13 18 2023-02-18 12:37:40.25037 +0800 CST m=+
0.019002401}
HandleRPC:&{true 2023-02-18 12:37:40.2493083 +0800 CST m=+0.017940701 2023-02-18 12:37:40.25037 +0800 CST m=+0.019002401 map[] <nil>}
2023/02/18 12:37:40 Greeting: Hello world

# 项目结构
$ tree demo/
demo/
├── client
│   └── client.go
├── go.mod
├── go.sum
├── pb
│   ├── helloword.pb.go
│   └── helloword.proto
└── server
    └── server.go

3 directories, 6 files

参考地址：https://github.com/grpc/grpc-go/blob/master/Documentation/keepalive.md