记录k8s kube-controller-manager-k8s-master kube-scheduler-k8s-master重启

1、报错如下

I0529 01:47:12.679312       1 event.go:307] "Event occurred" object="k8s-node-1" fieldPath="" kind="Node" apiVersion="v1" type="Normal" reason="CIDRNotAvailable" message="Node k8s-node-1 status is now: CIDRNotAvailable"
E0529 01:48:44.516760       1 controller_utils.go:262] Error while processing Node Add/Delete: failed to allocate cidr from cluster cidr at idx:0: CIDR allocation failed; there are no remaining CIDRs left to allocate in the accepted range
I0529 01:48:44.516885       1 event.go:307] "Event occurred" object="k8s-master" fieldPath="" kind="Node" apiVersion="v1" type="Normal" reason="CIDRNotAvailable" message="Node k8s-master status is now: CIDRNotAvailable"
E0529 01:49:28.020461       1 controller_utils.go:262] Error while processing Node Add/Delete: failed to allocate cidr from cluster cidr at idx:0: CIDR allocation failed; there are no remaining CIDRs left to allocate in the accepted range
I0529 01:49:28.020839       1 event.go:307] "Event occurred" object="k8s-node-2" fieldPath="" kind="Node" apiVersion="v1" type="Normal" reason="CIDRNotAvailable" message="Node k8s-node-2 status is now: CIDRNotAvailable"

2、可能原因k8s初始化cluster-cidr和service-cluster-ip-range子网划分冲突

（原    --apiserver-advertise-address 10.19.3.15  --service-cidr 10.245.0.0/12    --pod-network-cidr 10.244.0.0/16）

3、修改/etc/kubernetes/manifests/kube-controller-manager.yaml

- --cluster-cidr=10.96.0.0/16                #修改此处

修改后不用重启组件或者集群，稍等后pod自动重建

 kubectl logs  kube-controller-manager-k8s-master  -n kube-system查看没有报错

（未修改之前，即使pod运行正常，查看pod日志还是有上述报错）

4、修改后没有发现问题，请大家在测试环境测试。

///

更新------------------新建测试环境依然存在类似问题

centos7  kube-1.27.1  containerd 1.6.19 （master为esxi虚拟机）

集群状态都正常，系统硬盘IO也正常。依照网上的加大心跳时间、修改硬盘参数，都不能解决。

最后问题可能出现在Esxi这个集群的存储上，虚拟机迁移硬盘至另外的存储主机。因为目前已经24小时ks和kc组件未重启了，还不能最后排除，但是大概率就是这儿的问题。

日志如下，希望可以帮助需要的人：

1、containerd报错

Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.394405539+08:00" level=info msg="StopPodSandbox for \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\""
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.394537456+08:00" level=info msg="TearDown network for sandbox \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\" successfully"
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.394591107+08:00" level=info msg="StopPodSandbox for \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\" returns successfully"
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.395525689+08:00" level=info msg="RemovePodSandbox for \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\""
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.395780557+08:00" level=info msg="Forcibly stopping sandbox \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\""
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.396142546+08:00" level=info msg="TearDown network for sandbox \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\" successfully"
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.401661796+08:00" level=info msg="RemovePodSandbox \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\" returns successfully"
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.140016886+08:00" level=info msg="shim disconnected" id=8fd74e39f764d17344f0d5a0cfb92d6ade56421b0ce54d73d2971477d7a49eec
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.140641397+08:00" level=warning msg="cleaning up after shim disconnected" id=8fd74e39f764d17344f0d5a0cfb92d6ade56421b0ce54d73d2971477d7a49eec namespace=k8s.io
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.140835000+08:00" level=info msg="cleaning up dead shim"
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.152377826+08:00" level=warning msg="cleanup warnings time=\"2023-06-15T11:59:21+08:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=989621 runtime=io.containerd.runc.v2\n"
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.642769894+08:00" level=info msg="CreateContainer within sandbox \"2d27aa3e82f08d67ab9c6b8b821a324b86ba717b5d18c5729b382c488bd2f23f\" for container &ContainerMetadata{Name:kube-controller-manager,Attempt:1,}"
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.741636615+08:00" level=info msg="CreateContainer within sandbox \"2d27aa3e82f08d67ab9c6b8b821a324b86ba717b5d18c5729b382c488bd2f23f\" for &ContainerMetadata{Name:kube-controller-manager,Attempt:1,} returns container id \"53940ebcbf87d9313bb88b1656bc189745b04414888305a7ec47aef9c55fcdaf\""
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.742513549+08:00" level=info msg="StartContainer for \"53940ebcbf87d9313bb88b1656bc189745b04414888305a7ec47aef9c55fcdaf\""
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.855170471+08:00" level=info msg="StartContainer for \"53940ebcbf87d9313bb88b1656bc189745b04414888305a7ec47aef9c55fcdaf\" returns successfully"
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.392743400+08:00" level=info msg="shim disconnected" id=8eaef5a52f673266d0e141ae17a2d12ee377b7f08ad4a3f65d77f3abe0902c45
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.392850972+08:00" level=warning msg="cleaning up after shim disconnected" id=8eaef5a52f673266d0e141ae17a2d12ee377b7f08ad4a3f65d77f3abe0902c45 namespace=k8s.io
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.392869777+08:00" level=info msg="cleaning up dead shim"
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.405071189+08:00" level=warning msg="cleanup warnings time=\"2023-06-15T11:59:26+08:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=989732 runtime=io.containerd.runc.v2\n"
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.665183619+08:00" level=info msg="CreateContainer within sandbox \"24ebd020c4398151bdd87a97849fe02c4880f88bf132407e07ed5fad7c088932\" for container &ContainerMetadata{Name:kube-scheduler,Attempt:1,}"
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.719306795+08:00" level=info msg="CreateContainer within sandbox \"24ebd020c4398151bdd87a97849fe02c4880f88bf132407e07ed5fad7c088932\" for &ContainerMetadata{Name:kube-scheduler,Attempt:1,} returns container id \"e9f21ce36cace252a65544577beda1c6573e7473afc25d059df5d9234f18308b\""
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.720429540+08:00" level=info msg="StartContainer for \"e9f21ce36cace252a65544577beda1c6573e7473afc25d059df5d9234f18308b\""
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.834912477+08:00" level=info msg="StartContainer for \"e9f21ce36cace252a65544577beda1c6573e7473afc25d059df5d9234f18308b\" returns successfully"

2、etcd报错

{"level":"warn","ts":"2023-06-15T03:11:46.440Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"647.783649ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"info","ts":"2023-06-15T03:11:46.440Z","caller":"traceutil/trace.go:171","msg":"trace[1077021032] linearizableReadLoop","detail":"{readStateIndex:311409; appliedIndex:311408; }","duration":"393.007455ms","start":"2023-06-15T03:11:46.047Z","end":"2023-06-15T03:11:46.440Z","steps":["trace[1077021032] 'read index received'  (duration: 42.34µs)","trace[1077021032] 'applied index is now lower than readState.Index'  (duration: 392.963682ms)"],"step_count":2}
{"level":"warn","ts":"2023-06-15T03:11:46.440Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"393.171763ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"range_response_count:0 size:6"}
{"level":"info","ts":"2023-06-15T03:11:46.440Z","caller":"traceutil/trace.go:171","msg":"trace[2049818894] range","detail":"{range_begin:/registry/health; range_end:; response_count:0; response_revision:274023; }","duration":"393.252264ms","start":"2023-06-15T03:11:46.047Z","end":"2023-06-15T03:11:46.440Z","steps":["trace[2049818894] 'agreement among raft nodes before linearized reading'  (duration: 393.104647ms)"],"step_count":1}
{"level":"warn","ts":"2023-06-15T03:11:46.440Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-06-15T03:11:46.047Z","time spent":"393.363051ms","remote":"127.0.0.1:48410","response type":"/etcdserverpb.KV/Range","request count":0,"request size":18,"response count":0,"response size":30,"request content":"key:\"/registry/health\" "}
{"level":"info","ts":"2023-06-15T03:11:46.440Z","caller":"traceutil/trace.go:171","msg":"trace[499563569] transaction","detail":"{read_only:false; response_revision:274023; number_of_response:1; }","duration":"709.954696ms","start":"2023-06-15T03:11:45.730Z","end":"2023-06-15T03:11:46.440Z","steps":["trace[499563569] 'process raft request'  (duration: 61.343055ms)","trace[499563569] 'compare'  (duration: 647.534581ms)"],"step_count":2}
{"level":"warn","ts":"2023-06-15T03:11:46.440Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-06-15T03:11:45.730Z","time spent":"710.061185ms","remote":"127.0.0.1:48342","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":493,"response count":0,"response size":42,"request content":"compare: success:> failure: >"}
{"level":"info","ts":"2023-06-15T03:16:07.151Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":273955}
{"level":"info","ts":"2023-06-15T03:16:07.153Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":273955,"took":"1.552321ms","hash":1950790470}
{"level":"info","ts":"2023-06-15T03:16:07.153Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1950790470,"revision":273955,"compact-revision":273469}
{"level":"info","ts":"2023-06-15T03:20:43.437Z","caller":"traceutil/trace.go:171","msg":"trace[126699866] transaction","detail":"{read_only:false; response_revision:274895; number_of_response:1; }","duration":"307.20387ms","start":"2023-06-15T03:20:43.130Z","end":"2023-06-15T03:20:43.437Z","steps":["trace[126699866] 'process raft request'  (duration: 307.036766ms)"],"step_count":1}
{"level":"warn","ts":"2023-06-15T03:20:43.438Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-06-15T03:20:43.130Z","time spent":"307.376032ms","remote":"127.0.0.1:48342","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":465,"response count":0,"response size":42,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2023-06-15T03:21:05.415Z","caller":"etcdserver/v3_server.go:840","msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":1773161178989145684,"retry-timeout":"500ms"}