consul envoy upstream connect error or disconnect

upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

consul和envoy一起构建成的服务网格,上述的问题估计会经常发现,也头疼了我很久,还是总结列一下。

首先,应该从哪里开始检查?

1、看内容,就是tls的校验问题,根本不关你上游的服务什么事情。

2、要知道envoy之间,tls的控制来源是啥?consul!!!!!consul通过xds来动态配置envoy。所以,envoy出问题,很大程度是来源consul。不要单单看envoy的日志,其实感觉看envoy的日志,对于这种问题,感觉没啥卵用,直接去查一下,相关的envoy节点所关联的consul节点日志。

2023-06-26T11:25:35.211+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Sign server=10.70.51.58:47024 error="rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate"
2023-06-26T11:25:35.211+0800 [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-leaf error="rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate" index=0
2023-06-26T11:25:35.211+0800 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=ingress-gateway proxy=ingress-service-eai service_id=ingress-service-eai id=leaf error="error filling agent cache: rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate"
2023-06-26T11:26:38.201+0800 [ERROR] agent.client: RPC failed to server: method=ConnectCA.Sign server=10.70.51.58:47004 error="rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate"
2023-06-26T11:26:38.201+0800 [WARN]  agent.cache: handling error in Cache.Notify: cache-type=connect-ca-leaf error="rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate" index=0
2023-06-26T11:26:38.201+0800 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=ingress-gateway proxy=ingress-service-eai service_id=ingress-service-eai id=leaf error="error filling agent cache: rpc error making call: rpc error making call: CA is uninitialized and unable to sign certificates yet: no root certificate"

既然引期故障的日志找到了,就开始处理问题。

其实这个问题是因为consul connect 我采用的是vault,刚好昨天vault服务所在的服务器重启了,vault又没有配置自启,为啥又到了今天才报异常?因为刚好要更换leaf cert。consul感知到vault服务的异常,就所有的服务都gg了。

vault 这个蛋疼的服务,非必要,不要搞,老老实实用consul自带的cert。consul和envoy在国内的文档除了非常简单的,稍微深入一点的都没有,vault比它们更偏门,Google都不好使。

立刻把vault删了,把ca的配置改成用consul的,重启consul,完事

你可能感兴趣的:(错误解决,consul,envoy)