更新应用时,如何实现 K8s 零中断滚动更新?

Kubernetes 集群中,业务通常采用 Deployment + LoadBalancer 类型 Service 的方式对外提供服务,其典型部署架构如图 1 所示。这种架构部署和运维都十分简单方便,但是在应用更新或者升级时可能会存在服务中断,引发线上问题。今天我们来详细分析下这种架构为何在更新应用时会发生服务中断以及如何避免服务中断。
为何会发生服务中断
Deployment 滚动更新时会先创建新 pod,等待新 pod running 后再删除旧 pod。
新建 Pod
中断原因:Pod running 后被加入到 Endpoint 后端,容器服务监控到 Endpoint 变更后将 Node 加入到 SLB 后端。此时请求从 SLB 转发到 Pod 中,但是 Pod 业务代码还未初始化完毕,无法处理请求,导致服务中断,如图 2 所示。 解决方法:为 pod 配置就绪检测,等待业务代码初始化完毕后后再将 node 加入到 SLB 后端。
删除 Pod
在删除旧 pod 过程中需要对多个对象(如 Endpoint、ipvs/iptables、SLB)进行状态同步,并且这些同步操作是异步执行的,整体同步流程如图 3 所示。
pod 状态变更:将 Pod 设置为 Terminating 状态,并从所有 Service 的 Endpoints 列表中删除。此时,Pod 停止获得新的流量,但在 Pod 中运行的容器不会受到影响;
执行 preStop Hook:Pod 删除时会触发 preStop Hook,preStop Hook 支持 bash 脚本、TCP 或 HTTP 请求;
发送 SIGTERM 信号:向 Pod 中的容器发送 SIGTERM 信号;
等待指定的时间:terminationGracePeriodSeconds 字段用于控制等待时间,默认值为 30 秒。该步骤与 preStop Hook 同时执行,因此 terminationGracePeriodSeconds 需要大于 preStop 的时间,否则会出现 preStop 未执行完毕,pod 就被 kill 的情况;
发送 SIGKILL 信号:等待指定时间后,向 pod 中的容器发送 SIGKILL 信号,删除 pod。
中断原因:上述 1、2、3、4步骤同时进行,因此有可能存在 Pod 收到 SIGTERM 信号并且停止工作后,还未从 Endpoints 中移除的情况。此时,请求从 slb 转发到 pod 中,而 Pod 已经停止工作,因此会出现服务中断,如图 4 所示。

https://www.imdb.com/list/ls080196978/
https://www.imdb.com/list/ls080196967/
https://www.imdb.com/list/ls080196993/
https://www.imdb.com/list/ls080196806/
https://www.imdb.com/list/ls080196859/
https://www.imdb.com/list/ls080196819/
https://www.imdb.com/list/ls080196845/
https://www.imdb.com/list/ls080196891/
https://www.imdb.com/list/ls080192002/
https://www.imdb.com/list/ls080192077/
https://www.imdb.com/list/ls080192014/
https://www.imdb.com/list/ls080192034/
https://www.imdb.com/list/ls080192028/
https://www.imdb.com/list/ls080192091/
https://www.imdb.com/list/ls080192507/
https://www.imdb.com/list/ls080192574/
https://www.imdb.com/list/ls080192533/
https://www.imdb.com/list/ls080192522/
https://www.imdb.com/list/ls080192540/
https://www.imdb.com/list/ls080192596/
https://www.imdb.com/list/ls080192754/
https://www.imdb.com/list/ls080192778/
https://www.imdb.com/list/ls080192739/
https://www.imdb.com/list/ls080192721/
https://www.imdb.com/list/ls080192746/
https://www.imdb.com/list/ls080192780/
https://www.imdb.com/list/ls080192106/
https://www.imdb.com/list/ls080192177/
https://www.imdb.com/list/ls080192114/
https://www.imdb.com/list/ls080192127/
https://www.imdb.com/list/ls080192191/
https://www.imdb.com/list/ls080192304/
https://www.imdb.com/list/ls080192373/
https://www.imdb.com/list/ls080192319/
https://www.imdb.com/list/ls080192326/
https://www.imdb.com/list/ls080192397/
https://www.imdb.com/list/ls080192382/
https://www.imdb.com/list/ls080192655/
https://www.imdb.com/list/ls080192619/
https://www.imdb.com/list/ls080192632/
https://www.imdb.com/list/ls080192629/
https://www.imdb.com/list/ls080192685/
https://www.imdb.com/list/ls080192209/
https://www.imdb.com/list/ls080192277/
https://www.imdb.com/list/ls080192211/
https://www.imdb.com/list/ls080192235/
https://www.imdb.com/list/ls080192268/
https://www.imdb.com/list/ls080192241/
https://www.imdb.com/list/ls080192296/
https://www.imdb.com/list/ls080192450/
https://www.imdb.com/list/ls080192476/
https://www.imdb.com/list/ls080192433/
https://www.imdb.com/list/ls080192490/
https://www.imdb.com/list/ls080192904/
https://www.imdb.com/list/ls080192917/
https://www.imdb.com/list/ls080192921/
https://www.imdb.com/list/ls080192997/
https://www.imdb.com/list/ls080192807/
https://www.imdb.com/list/ls080192852/
https://www.imdb.com/list/ls080192813/
https://www.imdb.com/list/ls080192860/
https://www.imdb.com/list/ls080192823/
https://www.imdb.com/list/ls080192898/
https://www.imdb.com/list/ls080194001/
https://www.imdb.com/list/ls080194076/
https://www.imdb.com/list/ls080194036/
https://www.imdb.com/list/ls080194044/
https://www.imdb.com/list/ls080194082/
https://www.imdb.com/list/ls080194556/
https://www.imdb.com/list/ls080194519/
https://www.imdb.com/list/ls080194528/
https://www.imdb.com/list/ls080194599/
https://www.imdb.com/list/ls080194706/
https://www.imdb.com/list/ls080194771/
https://www.imdb.com/list/ls080194712/
https://www.imdb.com/list/ls080194764/
https://www.imdb.com/list/ls080194748/
https://www.imdb.com/list/ls080194786/
https://www.imdb.com/list/ls080194151/
https://www.imdb.com/list/ls080194111/
https://www.imdb.com/list/ls080194161/
https://www.imdb.com/list/ls080194143/
https://www.imdb.com/list/ls080194180/
https://www.imdb.com/list/ls080194357/
https://www.imdb.com/list/ls080194372/
https://www.imdb.com/list/ls080194368/
https://www.imdb.com/list/ls080194342/
https://www.imdb.com/list/ls080194386/
https://www.imdb.com/list/ls080194651/
https://www.imdb.com/list/ls080194612/
https://www.imdb.com/list/ls080194660/
https://www.imdb.com/list/ls080194629/
https://www.imdb.com/list/ls080194696/
https://www.imdb.com/list/ls080194682/
https://www.imdb.com/list/ls080194277/
https://www.imdb.com/list/ls080194237/
https://www.imdb.com/list/ls080194223/
https://www.imdb.com/list/ls080194249/
https://www.imdb.com/list/ls080194406/
https://www.imdb.com/list/ls080194476/
https://www.imdb.com/list/ls080194430/
https://www.imdb.com/list/ls080194434/
https://www.imdb.com/list/ls080194426/
https://www.imdb.com/list/ls080194497/
https://www.imdb.com/list/ls080194486/
https://www.imdb.com/list/ls080194956/
https://www.imdb.com/list/ls080194913/
https://www.imdb.com/list/ls080194938/
https://www.imdb.com/list/ls080194928/
https://www.imdb.com/list/ls080194993/
https://www.imdb.com/list/ls080194806/
https://www.imdb.com/list/ls080194810/
https://www.imdb.com/list/ls080194820/
https://www.imdb.com/list/ls080194848/
https://www.imdb.com/list/ls080194899/
https://www.imdb.com/list/ls080199052/
https://www.imdb.com/list/ls080199013/
https://www.imdb.com/list/ls080199020/
https://www.imdb.com/list/ls080199043/
https://www.imdb.com/list/ls080199096/
https://www.imdb.com/list/ls080193460/
https://www.imdb.com/list/ls080193429/
https://www.imdb.com/list/ls080193486/
https://www.imdb.com/list/ls080193904/
https://www.imdb.com/list/ls080193977/
https://www.imdb.com/list/ls080193914/
https://www.imdb.com/list/ls080193964/
https://www.imdb.com/list/ls080193924/
https://www.imdb.com/list/ls080193996/
https://www.imdb.com/list/ls080193981/
https://www.imdb.com/list/ls080193804/
https://www.imdb.com/list/ls080193873/
https://www.imdb.com/list/ls080193812/
https://www.imdb.com/list/ls080193831/
https://www.imdb.com/list/ls080193866/
https://www.imdb.com/list/ls080193845/
https://www.imdb.com/list/ls080193896/
https://www.imdb.com/list/ls080196007/
https://www.imdb.com/list/ls080196058/
https://www.imdb.com/list/ls080196078/
https://www.imdb.com/list/ls080196033/
https://www.imdb.com/list/ls080196027/
https://www.imdb.com/list/ls080196042/
https://www.imdb.com/list/ls080196503/
https://www.imdb.com/list/ls080196556/
https://www.imdb.com/list/ls080196513/
https://www.imdb.com/list/ls080196539/
https://www.imdb.com/list/ls080196521/
https://www.imdb.com/list/ls080196592/
https://www.imdb.com/list/ls080196709/
https://www.imdb.com/list/ls080196775/
https://www.imdb.com/list/ls080196735/
https://www.imdb.com/list/ls080196760/
https://www.imdb.com/list/ls080196722/
https://www.imdb.com/list/ls080196796/
https://www.imdb.com/list/ls080196103/
https://www.imdb.com/list/ls080196176/
https://www.imdb.com/list/ls080196133/
https://www.imdb.com/list/ls080196123/
https://www.imdb.com/list/ls080196149/
https://www.imdb.com/list/ls080196306/
https://www.imdb.com/list/ls080196359/
https://www.imdb.com/list/ls080196315/
https://www.imdb.com/list/ls080196333/
https://www.imdb.com/list/ls080196365/
https://www.imdb.com/list/ls080196343/
https://www.imdb.com/list/ls080196387/
https://www.imdb.com/list/ls080196608/
https://www.imdb.com/list/ls080196678/
https://www.imdb.com/list/ls080196639/

你可能感兴趣的:(更新应用时,如何实现 K8s 零中断滚动更新?)