大规模场景下对Istio的性能优化
简介
当前istio下发xDS使用的是全量下发策略，也就是网格里的所有sidecar(envoy)，内存里都会有整个网格内所有的服务发现数据。这样的结果是，每个sidecar内存都会随着网格规模增长而增长。
Aeraki-mesh
aeraki-mesh项目下有一个子项目专门用来处理istio配置分发性能问题，我们找到该项目：
https://github.com/aeraki-mesh/lazyxds
从该项目的部署yaml中，我们知道它会在网格中增加两个组件：
egress：充当类似网格模型中默认网关角色
controller：用来分析并补全服务间的依赖关系
Egress
对应的配置文件为：lazyxds-egress.yaml
下面来一一查看该组件的组成部分
组件配置
apiVersion: apps/v1
kind: Deployment
metadata:
name: istio-egressgateway-lazyxds
namespace: istio-system
labels:
app: istio-egressgateway-lazyxds
istio: egressgateway
spec:
replicas: 1
selector:
matchLabels:
app: istio-egressgateway-lazyxds
istio: egressgateway
template:
metadata:
annotations:
sidecar.istio.io/discoveryAddress: istiod.istio-system.svc:15012
sidecar.istio.io/inject: "false"
labels:
app: istio-egressgateway-lazyxds
istio: egressgateway
spec:
containers:
- args:
......
image: docker.io/istio/proxyv2:1.10.0
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
......
volumeMounts:
- mountPath: /etc/istio/custom-bootstrap
name: custom-bootstrap-volume
......
volumes:
- configMap:
defaultMode: 420
name: lazyxds-als-bootstrap
name: custom-bootstrap-volume由于配置太多，这里只挑选主要的部分，从上面可以看出，其实是启动一个istio proxy，该proxy的启动配置文件是使用的configmap挂载出来的。
启动配置
apiVersion: v1
kind: ConfigMap
metadata:
name: lazyxds-als-bootstrap
namespace: istio-system
data:
custom_bootstrap.json: |
{
"static_resources": {
"clusters": [{
"name": "lazyxds-accesslog-service",
"type": "STRICT_DNS",
"connect_timeout": "1s",
"http2_protocol_options": {},
"dns_lookup_family": "V4_ONLY",
"load_assignment": {
"cluster_name": "lazyxds-accesslog-service",
"endpoints": [{
"lb_endpoints": [{
"endpoint": {
"address": {
"socket_address": {
"address": "lazyxds.istio-system",
"port_value": 8080
}
}
}
}]
}]
},
"respect_dns_ttl": true
}]
}
}从上面配置可以知道：
定义了proxy组件代理的集群，该集群为
"lazyxds-accesslog-service"该集群对应的后端服务地址为
"lazyxds.istio-system"，端口为8080
这个后端就是lazyxds controller，后面细说
EnvoyFilter
从yaml文件我们看到，还定义了一个envoyfilter来修改proxy代理的流量配置
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: lazyxds-egress-als
namespace: istio-system
spec:
workloadSelector:
labels:
app: istio-egressgateway-lazyxds
configPatches:
- applyTo: NETWORK_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.filters.network.http_connection_manager"
patch:
operation: MERGE
value:
typed_config:
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
access_log:
......
- name: envoy.access_loggers.http_grpc
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.grpc.v3.HttpGrpcAccessLogConfig
common_config:
log_name: http_envoy_accesslog
transport_api_version: "V3"
grpc_service:
envoy_grpc:
cluster_name: lazyxds-accesslog-service从这个配置文件，可以看出在启动envoy时，会向其注入一个accesslog service，也就是envoy的日志收集器，而这个service就是lazyxds-accesslog-service
Controller
具体的lazy xds实现就是通过这个controller实现的
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: lazyxds
name: lazyxds
namespace: istio-system
spec:
replicas: 1
selector:
matchLabels:
app: lazyxds
template:
metadata:
labels:
app: lazyxds
spec:
serviceAccountName: lazyxds
containers:
- image: aeraki/lazyxds:latest
imagePullPolicy: Always
name: app
ports:
- containerPort: 8080
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app: lazyxds
name: lazyxds
namespace: istio-system
spec:
ports:
- name: grpc-als
port: 8080
protocol: TCP
selector:
app: lazyxds
type: ClusterIP从配置可以看到，在egress环节我们知道了proxy的代理的后端地址为lazyxds.istio-system，刚好对应这里的controller。
并且我们还知道，envoy的访问日志最终会发送给这个controller来处理，而这就是实现增量下发envoy配置的关键之处，也就是解决istio性能的解决之法。
增量下发
Accesslog接口
要接受envoy的访问日志，必须实现envoy定义的接口：
type AccessLogServiceServer interface {
// Envoy will connect and send StreamAccessLogsMessage messages forever. It does not expect any
// response to be sent as nothing would be done in the case of failure. The server should
// disconnect if it expects Envoy to reconnect. In the future we may decide to add a different
// API for "critical" access logs in which Envoy will buffer access logs for some period of time
// until it gets an ACK so it could then retry. This API is designed for high throughput with the
// expectation that it might be lossy.
StreamAccessLogs(AccessLogService_StreamAccessLogsServer) error
}日志解析
lazyxds实现如下：
func (server *Server) StreamAccessLogs(logStream als.AccessLogService_StreamAccessLogsServer) error {
for {
data, err := logStream.Recv()
if err != nil {
return err
}
httpLog := data.GetHttpLogs()
if httpLog != nil {
for _, entry := range httpLog.LogEntry {
server.log.V(4).Info("http log entry", "entry", entry)
fromIP := getDownstreamIP(entry)
if fromIP == "" {
continue
}
upstreamCluster := entry.CommonProperties.UpstreamCluster
svcID := utils.UpstreamCluster2ServiceID(upstreamCluster)
toIP := getUpstreamIP(entry)
if err := server.handler.HandleAccess(fromIP, svcID, toIP); err != nil {
server.log.Error(err, "handle access error")
}
}
}
}
}上面主要的逻辑就是解析envoy的访问日志，然后进行处理：
lazy xds Controller 会对接收到的日志进行访问关系分析，然后把新的依赖关系表达到 sidecar CRD 中。
同时 Controller 还会更新 Egress 的规则：删除、更新或创建。
Slime
网易Slime方案与腾讯云Aeraki方案的思路一致
文档：https://cloudnative.to/blog/netease-slime/
github：https://github.com/slime-io/slime/tree/master/staging/src/slime.io/slime/modules/lazyload
https://cloud.tencent.com/developer/article/1922778
https://www.zhaohuabing.com/post/2018-09-25-istio-traffic-management-impl-intro/