故障概述
接到反馈有kubelet的10250端口无法连接，经过排查发现有部分Node是NotReady状态，且很多Pod是Terminating状态，此时整个K8S集群都处于失控状态，业务不能正常工作，老的Pod不断被删除，新的Pod又很难被调度，又有部分Pod无法删除。压测的部分业务无法正常工作。
故障分析
为什么出现很多NotReady？
查看Node状态，此时已经有部分Node的状态为NotReady
kubectl get nodes
NAME STATUS ROLES AGE VERSION
op-k8s1-pm Ready 66d v1.17.3
op-k8s10-pm Ready 46h v1.17.3
op-k8s11-pm Ready 46h v1.17.3
op-k8s2-pm NotReady 66d v1.17.3
op-k8s3-pm NotReady 66d v1.17.3
op-k8s4-pm NotReady 66d v1.17.3
op-k8s5-pm NotReady 66d v1.17.3
op-k8s6-pm NotReady 66d v1.17.3
...
op-k8smaster3-pm Ready master 69d v1.17.3
以下为op-k8s2-pm上的资源使用情况排查
free -g
total used free shared buff/cache available
Mem: 250 242 1 0 7 3
Swap: 0 0 0
uptime
18:10:11 up 70 days, 8 min, 2 users, load average: 733.31, 616.92, 625.68
ps aux|grep java |grep -v tini |wc -l
91
top
top - 18:11:24 up 70 days, 10 min, 2 users, load average: 579.80, 607.49, 622.64
Tasks: 1069 total, 3 running, 688 sleeping, 0 stopped, 0 zombie
%Cpu(s): 9.4 us, 7.0 sy, 0.0 ni, 0.2 id, 81.5 wa, 0.4 hi, 1.5 si, 0.0 st
KiB Mem : 26275092+total, 984572 free, 25881926+used, 2947100 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 370160 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
973481 nfsnobo+ 20 0 126808 28788 0 S 403.5 0.0 709:41.85 node_exporter
957773 1004 20 0 16.6g 3.0g 0 S 100.0 1.2 3:02.27 java
277 root 20 0 0 0 0 R 99.1 0.0 52:00.89 kswapd0
278 root 20 0 0 0 0 R 99.1 0.0 100:21.86 kswapd1
895608 root 20 0 1706728 3600 2336 D 7.1 0.0 3:04.87 journalctl
874 root 20 0 4570848 127852 0 S 4.4 0.0 8:16.75 kubelet
11115 maintain 20 0 165172 1760 0 R 2.7 0.0 0:00.21 top
965470 1004 20 0 17.3g 2.8g 0 S 2.7 1.1 1:59.22 java
9838 root 20 0 0 0 0 I 1.8 0.0 0:04.95 kworker/u98:0-f
952613 1004 20 0 19.7g 2.8g 0 S 1.8 1.1 1:51.01 java
954967 1004 20 0 13.6g 3.0g 0 S 1.8 1.2 3:00.73 java
此时在op-k8s2-pm已经出现大量的Terminating的Pod
kubectl get pod -owide --all-namespaces |grep op-k8s2-pm
test tutor-episode--venv-stress-fudao-6c68ff7f89-w49tv 1/1 Terminating 0 23h 10.1.4.56 op-k8s2-pm
test tutor-es-lesson--venv-stress-fudao-69f67c4dc4-r56m4 1/1 Terminating 0 23h 10.1.4.93 op-k8s2-pm
test tutor-faculty--venv-stress-fudao-7f44fbdcd5-dzcxq 1/1 Terminating 0 23h 10.1.4.45 op-k8s2-pm
...
test tutor-oauth--venv-stress-fudao-5989489c9d-jtzgg 1/1 Terminating 0 23h 10.1.4.78 op-k8s2-pm
kubelet日志，出现了很多“use of closed network connection”， 由于K8S的默认连接是HTTP2.0长连接，此错误表明此kubelet连接到Apiserver的连接是broken的，在Apiserver上通过netstat观察也没有45688这个连接存在
Nov 05 21:58:35 op-k8s2-pm kubelet[105611]: E1105 21:58:35.276562 105611 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.RuntimeClass: Get https://10.2.2.2:443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: write tcp 10.2.2.7:45688->10.2.2.2:443: use of closed network connection
Nov 05 21:58:35 op-k8s2-pm kubelet[105611]: I1105 21:58:35.346898 105611 config.go:100] Looking for [api file], have seen map[]
Nov 05 21:58:35 op-k8s2-pm kubelet[105611]: I1105 21:58:35.446871 105611 config.go:100] Looking for [api file], have seen map[]
到此时基本判定soho-op-k8s2-pm的资源耗尽导致kubelet无法连接Apiserver，最终出现NotRead。经过排查其他的Node，情况基本一致。
为什么突然出现这么大量的资源争抢？
tutor-lesson-activity--venv-stress-fudao 16/16 16 16 12h28m
tutor-lesson-renew--venv-stress-fudao 9/10 9 32 12h45m
tutor-live-data-check 9/32 9 32 12h10m
tutor-oauth--venv-stress-fudao 16/32 16 32 12h19m
tutor-pepl--venv-stress-fudao 10/32 10 32 12h
tutor-profile--venv-stress-fudao 7/32 7 32 12h
tutor-recommend--venv-stress-fudao 32/32 32 32 12h
...
tutor-student-lesson--venv-stress-fudao 24/24 24 24 12h
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
yfd_deploy_version: 875f2832-1e81-11eb-ba04-00163e0a5041
labels:
project: tutor-recommend
name: tutor-recommend--venv-stress-fudao
namespace: test
spec:
replicas: 32
selector:
matchLabels:
project: tutor-recommend
template:
metadata:
labels:
project: tutor-recommen
spec:
containers:
resources:
limits:
cpu: "4"
memory: 8G
requests:
cpu: 500m
memory: 512M
经过询问是由于压测想用k8s测试环境模拟线上环境压测一下，但是测试节点数远远低于线上的节点数
结论：由于很多deployment增加了大量副本数,且资源超卖验证，最终导致K8S集群资源不足，造成了资源争抢，不断压垮kubelet，出现NotReady
为什么出现很多Terminating?
没有触发Evict啊(Evict的Pod的状态为Evict），而且Pod的状态Terminating，在排查的过程也发现不断有Pod生成，老的Pod不断Terminating。
为什么没有触发Evict呢？经过排查发现磁盘没有压力，内存的必须要小于100M才会触发Evict。以下的kubelet的配置就可能看出没有触发Evict
evictionHard:
imagefs.available: 1%
memory.available: 100Mi
nodefs.inodesFree: 1%
注一：由于Evict不可控，且一旦发现磁盘或者内存有压力，最好通过手动处理，自动处理很难解决问题，且很容出现迁移了一个Pod后节点恢复，但是一会儿又出现压力，来回来去触发Evict
注二：将来会彻底禁止Evict
排查kube-controller-manager的日志发现
endpoints_controller.go:590] Pod is out of service: test/tutor-episode--venv-stress-fudao-6c68ff7f89-h8wwz
taint_manager.go:105] NoExecuteTaintManager is deleting Pod: test/tutor-episode--venv-stress-fudao-6c68ff7f89-h8wwz
request.go:565] Throttling request took 1.148923424s, request: DELETE:https://10.2.2.2:443/api/v1/namespaces/test/pods/tutor-episode--venv-stress-fudao-6c68ff7f89-h8wwz
disruption.go:457] No PodDisruptionBudgets found for pod tutor-episode--venv-stress-fudao-6c68ff7f89-h8wwz, PodDisruptionBudget controller will avoid syncing
endpoints_controller.go:420] Pod is being deleted test/tutor-episode--venv-stress-fudao-6c68ff7f89-h8wwz
controller_utils.go:911] Ignoring inactive pod test/tutor-episode--venv-stress-fudao-6c68ff7f89-h8wwz in state Running, deletion time 2020-11-05 10:06:27 +0000 UTC
发现是由于NoExecuteTaintManager控制器删除的Pod，经过追查代码发现
kubernetes/pkg/controller/nodelifecycle/scheduler/taint_manager.go
handleNodeUpdate
func (tc *NoExecuteTaintManager) handleNodeUpdate(nodeUpdate nodeUpdateItem) {
node, err := tc.getNode(nodeUpdate.nodeName)
if err != nil {
if apierrors.IsNotFound(err) {
// Delete
klog.V(4).Infof("Noticed node deletion: %#v", nodeUpdate.nodeName)
tc.taintedNodesLock.Lock()
defer tc.taintedNodesLock.Unlock()
delete(tc.taintedNodes, nodeUpdate.nodeName)
return
}
utilruntime.HandleError(fmt.Errorf("cannot get node %s: %v", nodeUpdate.nodeName, err))
return
}
// Create or Update
klog.V(4).Infof("Noticed node update: %#v", nodeUpdate)
taints := getNoExecuteTaints(node.Spec.Taints)
func() {
tc.taintedNodesLock.Lock()
defer tc.taintedNodesLock.Unlock()
klog.V(4).Infof("Updating known taints on node %v: %v", node.Name, taints)
if len(taints) == 0 {
delete(tc.taintedNodes, node.Name)
} else {
tc.taintedNodes[node.Name] = taints
}
}()
// This is critical that we update tc.taintedNodes before we call getPodsAssignedToNode:
// getPodsAssignedToNode can be delayed as long as all future updates to pods will call
// tc.PodUpdated which will use tc.taintedNodes to potentially delete delayed pods.
pods, err := tc.getPodsAssignedToNode(node.Name)
if err != nil {
klog.Errorf(err.Error())
return
}
if len(pods) == 0 {
return
}
// Short circuit, to make this controller a bit faster.
if len(taints) == 0 {
klog.V(4).Infof("All taints were removed from the Node %v. Cancelling all evictions...", node.Name)
for i := range pods {
tc.cancelWorkWithEvent(types.NamespacedName{Namespace: pods[i].Namespace, Name: pods[i].Name})
}
return
}
now := time.Now()
for _, pod := range pods {
podNamespacedName := types.NamespacedName{Namespace: pod.Namespace, Name: pod.Name}
tc.processPodOnNode(podNamespacedName, node.Name, pod.Spec.Tolerations, taints, now)
}
}
processPodOnNode
func (tc *NoExecuteTaintManager) processPodOnNode(
podNamespacedName types.NamespacedName,
nodeName string,
tolerations []v1.Toleration,
taints []v1.Taint,
now time.Time,
) {
if len(taints) == 0 {
tc.cancelWorkWithEvent(podNamespacedName)
}
allTolerated, usedTolerations := v1helper.GetMatchingTolerations(taints, tolerations)
if !allTolerated {
klog.V(2).Infof("Not all taints are tolerated after update for Pod %v on %v", podNamespacedName.String(), nodeName)
// We're canceling scheduled work (if any), as we're going to delete the Pod right away.
tc.cancelWorkWithEvent(podNamespacedName)
tc.taintEvictionQueue.AddWork(NewWorkArgs(podNamespacedName.Name, podNamespacedName.Namespace), time.Now(), time.Now())
return
}
minTolerationTime := getMinTolerationTime(usedTolerations)
// getMinTolerationTime returns negative value to denote infinite toleration.
if minTolerationTime < 0 {
klog.V(4).Infof("New tolerations for %v tolerate forever. Scheduled deletion won't be cancelled if already scheduled.", podNamespacedName.String())
return
}
startTime := now
triggerTime := startTime.Add(minTolerationTime)
scheduledEviction := tc.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String())
if scheduledEviction != nil {
startTime = scheduledEviction.CreatedAt
if startTime.Add(minTolerationTime).Before(triggerTime) {
return
}
tc.cancelWorkWithEvent(podNamespacedName)
}
tc.taintEvictionQueue.AddWork(NewWorkArgs(podNamespacedName.Name, podNamespacedName.Namespace), startTime, triggerTime)
}
发现LifecycleController控制会监控所有的Node，一旦发现Node的taint有NoExecuted，就检查其上的Pod是否有toleration，如果没有或者TolerationSeconds时间小于0，就启动Pod删除动作。
kubernetes的文档上的解释 (详见：https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
)
NoExecute
Normally, if a taint with effect NoExecute is added to a node, then any pods that do not tolerate the taint will be evicted immediately, and pods that do tolerate the taint will never be evicted. However, a toleration with NoExecute effect can specify an optional tolerationSeconds field that dictates how long the pod will stay bound to the node after the taint is added.
kubectl get pods -n test tutor-recommend--venv-stress-fudao-5d89fc7dd5-vcp8h -ojson |jq '.spec.tolerations'
[
{
"effect": "NoExecute",
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"tolerationSeconds": 300
},
{
"effect": "NoExecute",
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"tolerationSeconds": 300
}
]
Pod上果然有设置not-ready和tolerationSeconds，但是这些tolerations谁设置呢？什么时候设置的？
从代码里发现，是由defaulttolerationseconds的adminssion controller设置的，是在Pod生成的时候默认就是设置的。 从v1.13以后的版本中，默认会启动此设置。用来替代--pod-eviction-timeout参数，来实现Evict的功能。
kubernetes/plugin/pkg/admission/defaulttolerationseconds/admission.go
Admit
// Admit makes an admission decision based on the request attributes
func (p *Plugin) Admit(ctx context.Context, attributes admission.Attributes, o admission.ObjectInterfaces) (err error) {
if attributes.GetResource().GroupResource() != api.Resource("pods") {
return nil
}
if len(attributes.GetSubresource()) > 0 {
// only run the checks below on pods proper and not subresources
return nil
}
pod, ok := attributes.GetObject().(*api.Pod)
if !ok {
return errors.NewBadRequest(fmt.Sprintf("expected *api.Pod but got %T", attributes.GetObject()))
}
tolerations := pod.Spec.Tolerations
toleratesNodeNotReady := false
toleratesNodeUnreachable := false
for _, toleration := range tolerations {
if (toleration.Key == v1.TaintNodeNotReady || len(toleration.Key) == 0) &&
(toleration.Effect == api.TaintEffectNoExecute || len(toleration.Effect) == 0) {
toleratesNodeNotReady = true
}
if (toleration.Key == v1.TaintNodeUnreachable || len(toleration.Key) == 0) &&
(toleration.Effect == api.TaintEffectNoExecute || len(toleration.Effect) == 0) {
toleratesNodeUnreachable = true
}
}
if !toleratesNodeNotReady {
pod.Spec.Tolerations = append(pod.Spec.Tolerations, notReadyToleration)
}
if !toleratesNodeUnreachable {
pod.Spec.Tolerations = append(pod.Spec.Tolerations, unreachableToleration)
}
return nil
}
结论
出现很多的Terminating的Pod是由于节点Node NotReady后触发NoExecute驱逐导致。在Deployment中可以使用功能，但是在statefulset中建议要取消此功能。