istio部署grpc服务时遇到的问题

个人博客原文地址：http://www.lampnick.com/php/916

问题一：mycaller服务请求myresponser服务时，http接口可以通，grpc服务不通

• 使用sleep容器对mycaller服务进行请求

 # kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
 {"code":10000000,"error":"内部错误","detail":"grpc: error while marshaling: proto: Marshal called with nil"}

返回错误，于是查看mycaller服务容器的日志

[root@master /root]# kubectl logs -f mycaller-6659dc46b-h4jzp mycaller
2020/02/28 22:07:49 [info] Start http server listen :59130
mycaller请求==>1
请求myresponser服务连接信息: {"ServerName":"myresponser","ServerPort":"53606","CertFile":"","CertServerName":"","IsSsl":false}
mycaller get请求url==>http://myresponser:53605/testResponser/GetHello?type=1
mycaller get请求fileBody==>{"res":"[2020-02-28 22:08:47] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:"}
请求myresponser服务客户端-获取Hello失败 err: rpc error: code = Unimplemented desc = Not Found: HTTP status code 404; transport: received the unexpected content-type "text/plain; charset=utf-8"
ERROR: 2020/02/28 22:08:47 grpc: server failed to encode response:  rpc error: code = Internal desc = grpc: error while marshaling: proto: Marshal called with nil
{"level":"error","caller":"/usr/local/go/src/git.myscrm.cn/trade/mycaller/vendor/git.myscrm.cn/golang/common/yklog/logger_context.go:202","time":"2020-02-28 22:08:47.084","msg":"grpc-gateway err: grpc: error while marshaling: proto: Marshal called with nil"}

看日志通过http get请求返回了正常的数据，但是grpc调用Hello方法时报错：rpc error: code = Unimplemented desc = Not Found: HTTP status code 404; transport: received the unexpected content-type “text/plain; charset=utf-8”，于是再查看mycaller pod中的envoy日志

2020-02-28T14:07:32.934346Z	info	Envoy proxy is ready
[2020-02-28T14:07:49.471Z] "- - -" 0 - "-" "-" 136 566 5 - "-" "-" "-" "-" "10.10.3.61:18011" PassthroughCluster 10.36.0.10:49416 10.10.3.61:18011 10.36.0.10:49414 - -
[2020-02-28T14:07:49.483Z] "GET /v2/keys/?quorum=false&recursive=false&sorted=false HTTP/1.1" 200 - "-" "-" 0 26283 2 1 "-" "Go-http-client/1.1" "45b99d06-d24a-4a9e-af17-430b582976c6" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - -
[2020-02-28T14:07:49.486Z] "PUT /v2/keys/service.mycaller.default HTTP/1.1" 200 - "-" "-" 137 413 0 0 "-" "Go-http-client/1.1" "511e9a64-8294-4fa9-97db-2710474c6816" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - -
[2020-02-28T14:08:47.063Z] "GET /v2/keys/service.myresponser.default?quorum=false&recursive=false&sorted=false HTTP/1.1" 200 - "-" "-" 0 216 1 1 "-" "Go-http-client/1.1" "aeb99ac6-17bf-4aea-8f6b-b58b6f3d49c8" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - -
[2020-02-28T14:08:47.066Z] "GET /testResponser/GetHello?type=1 HTTP/1.1" 200 - "-" "-" 0 113 9 9 "-" "Go-http-client/1.1" "82b0a59f-223c-4944-b630-59d2c728d059" "myresponser:53605" "10.36.0.11:53606" outbound|53605||myresponser.default.svc.cluster.local - 10.97.218.153:53605 10.36.0.10:35778 - default
[2020-02-28T14:08:47.077Z] "POST /testResponser.TestService/GetHello HTTP/2" 404 - "-" "-" 8 10 6 5 "-" "grpc-go/1.26.0" "be930121-1b34-4697-a909-3ec80dccbf18" "myresponser:53606" "10.36.0.11:53606" outbound|53606||myresponser.default.svc.cluster.local - 10.97.218.153:53606 10.36.0.10:37922 - default
[2020-02-28T14:08:47.060Z] "GET /testCaller/GetHello?type=1&orgcode=private&port=53605 HTTP/2" 500 - "-" "-" 0 112 24 23 "-" "curl/7.64.0" "744ba535-8d37-4fa4-8e87-2a7e755c432a" "mycaller.default:59130" "127.0.0.1:59130" inbound|59130|http2-59130|mycaller.default.svc.cluster.local - 10.36.0.10:59130 10.36.0.12:52096 outbound_.59130_.v1_.mycaller.default.svc.cluster.local default

发现是HTTP/2协议报的404，于是查看svc，是配置的grpc协议，不是http2

[root@master /root]# kubectl get svc myresponser -o yaml
apiVersion: v1
kind: Service
metadata:
  ...
  labels:
    app: myresponser
  name: myresponser
  namespace: default
  resourceVersion: "1982215"
  selfLink: /api/v1/namespaces/default/services/myresponser
  uid: 6fbd7ac2-0895-4d5c-b99f-2cdb6fc07e0d
spec:
  clusterIP: 10.97.218.153
  ports:
  - name: http-53605
    port: 53605
    protocol: TCP
    targetPort: 53606
  - name: grpc-53606
    port: 53606
    protocol: TCP
    targetPort: 53606
  selector:
    app: myresponser
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

于是修改svc，将name从grpc-53606修改为http2-53606

apiVersion: v1
kind: Service
metadata:
  name: myresponser
  namespace: default
  labels:
    app: myresponser
spec:
  selector:
    app: myresponser
  ports:
  - name: http-53605
    port: 53605
    protocol: TCP
    targetPort: 53606
  - name: http2-53606
    port: 53606
    protocol: TCP
    targetPort: 53606

再次请求mycaller服务，发现还是报一样的错误。搞了半天，索性把http部分去掉，只保留http2的port试试

apiVersion: v1
kind: Service
metadata:
  name: myresponser
  namespace: default
  labels:
    app: myresponser
spec:
  selector:
    app: myresponser
  ports:
#  - name: http-53605
#    port: 53605
#    protocol: TCP
#    targetPort: 53606
  - name: http2-53606
    port: 53606
    protocol: TCP
    targetPort: 53606

再请求mycaller服务，发现grpc请求成功了

# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
{"res":"[2020-02-28 22:30:31] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}

mycaller pod日志, 由于取消了http的端口，所以http请求不通，grpc正常

mycaller请求==>1
请求myresponser服务连接信息: {"ServerName":"myresponser","ServerPort":"53606","CertFile":"","CertServerName":"","IsSsl":false}
mycaller get请求url==>http://myresponser:53605/testResponser/GetHello?type=1
mycaller get请求Do err:==> Get http://myresponser:53605/testResponser/GetHello?type=1: read tcp 10.36.0.10:46620->10.97.218.153:53605: read: connection reset by peer
[2020-02-28 22:30:29] mycaller请求==> hostname:mycaller-6659dc46b-h4jzp, req:1, myresponser响应==>[2020-02-28 22:30:31] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private

mycaller中envoy的日志

[2020-02-28T14:30:29.863Z] "GET /v2/keys/service.myresponser.default?quorum=false&recursive=false&sorted=false HTTP/1.1" 200 - "-" "-" 0 216 1 1 "-" "Go-http-client/1.1" "0b71ef78-1cbc-4f40-a731-153d0833637d" "10.10.8.33:2379" "10.10.8.33:2379" PassthroughCluster - 10.10.8.33:2379 10.36.0.10:45622 - -
[2020-02-28T14:30:29.866Z] "- - -" 0 UF,URX "-" "-" 0 0 1000 - "-" "-" "-" "-" "10.97.218.153:53605" PassthroughCluster - 10.97.218.153:53605 10.36.0.10:46620 - -
[2020-02-28T14:30:30.866Z] "POST /testResponser.TestService/GetHello HTTP/2" 200 - "-" "-" 8 117 301 300 "-" "grpc-go/1.26.0" "ea990043-c6f7-416b-8aeb-d1c7b59844ba" "myresponser:53606" "10.5.24.224:32483" outbound|53606||myresponser.default.svc.cluster.local - 10.97.218.153:53606 10.36.0.10:48764 - default
[2020-02-28T14:30:29.861Z] "GET /testCaller/GetHello?type=1&orgcode=private&port=53605 HTTP/2" 200 - "-" "-" 0 120 1307 1307 "-" "curl/7.64.0" "2ea39954-58cc-4a32-8db9-96a0ed1c160b" "mycaller.default:59130" "127.0.0.1:59130" inbound|59130|http2-59130|mycaller.default.svc.cluster.local - 10.36.0.10:59130 10.36.0.12:52096 outbound_.59130_.v1_.mycaller.default.svc.cluster.local default

• 参考文档
  • https://istio.io/docs/ops/deployment/requirements/

问题二：当grpc通了之后，使用istio的VirtualService进行分流的时候出现流量没有按照规则进行

• 当时的VirtualService规则如下

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: myresponser-virtualservice
spec:
  hosts:
    - myresponser
  gateways:
  - myresponser-gateway
  http:
  - match:
    - headers:
        orgcode:
          exact: private 
    route:
    - destination:
        host: myresponser
        subset: v2
  - route:
    - destination:
        host: myresponser
        subset: v1

请求及响应如下

[root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
{"res":"[2020-03-02 10:57:55] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}[root@master /root/yunke-istio-manifests-problem]# 
[root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
{"res":"[2020-03-02 10:58:02] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}
[root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kupe=1&orgcode=private&port=53605"sonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?typ
{"res":"[2020-03-02 10:58:06] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}
[root@master /root/yunke-istio-manifests-problem]# kubectl exec -it -c sleep $(kubecpe=1&orgcode=private&port=53605"path='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?typ
{"res":"[2020-03-02 10:58:10] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:private"}[root@master /root/yunke-istio-manifests-problem]# 

经过排查，发现上面的yaml文件多配置了网关，后来把网关去掉，问题解决

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: myresponser-virtualservice
spec:
  hosts:
    - myresponser
#  gateways:
#  - myresponser-gateway
  http:
  - match:
    - headers:
        orgcode:
          exact: private 
    route:
    - destination:
        host: myresponser
        subset: v2
  - route:
    - destination:
        host: myresponser
        subset: v1

请求如下：

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
{"res":"[2020-03-03 09:09:59] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
{"res":"[2020-03-03 09:10:04] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=private&port=53605"
{"res":"[2020-03-03 09:10:07] responser version:private, hostname:myresponser-v2-67d7f6d7f4-6fctl, req:1, orgcode:private"}

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=&port=53605"
{"res":"[2020-03-03 09:10:16] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:prod"}

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=ttt&port=53605"
{"res":"[2020-03-03 09:10:23] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:ttt"}

[root@master /root]# kubectl exec -it -c sleep $(kubectl get pod  -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl "mycaller.default:59130/testCaller/GetHello?type=1&orgcode=prod&port=53605"
{"res":"[2020-03-03 09:10:30] responser version:prod, hostname:myresponser-v1-6b9f79c64d-fwl79, req:1, orgcode:prod"}

• 参考文档
  • https://istio.io/docs/tasks/traffic-management/traffic-shifting/