ingress-nginx部署状态为CrashLoopBackOff 问题排查

ingress-nginx部署状态为CrashLoopBackOff 问题排查，说起来这个问题挺坑的，kubernetes集群部署的时候，没有开启ipvs转发，所以会有这个问题

系统版本：CentOs8.1
k8s版本：v1.21.3
containerd版本：ctr containerd.io 1.4.3

ingress部署后状态为CrashLoopBackOff ,不停的重启

[root@ck8s1 ingress]# kubectl get pod -n ingress-nginx
NAME                                        READY   STATUS             RESTARTS   AGE
default-http-backend-7b6d9847f6-crgs8       1/1     Running            0          46m
nginx-ingress-controller-7bbb744996-rd9d6   0/1     CrashLoopBackOff   19         46m

describe查看信息如下：

[root@ck8s1 ingress]# kubectl describe pod nginx-ingress-controller-7bbb744996-rd9d6 -n ingress-nginx
Name:         nginx-ingress-controller-7bbb744996-rd9d6
Namespace:    ingress-nginx
Priority:     0
Node:         ck8s2/192.168.43.152
Start Time:   Sun, 15 Aug 2021 12:06:08 +0800
Labels:       app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              pod-template-hash=7bbb744996
Annotations:  cni.projectcalico.org/containerID: f59aa02e0654127673fc968533873d70d3b172c6f9cea0f7cab027efc662ee46
              cni.projectcalico.org/podIP: 192.168.135.151/32
              cni.projectcalico.org/podIPs: 192.168.135.151/32
              prometheus.io/port: 10254
              prometheus.io/scrape: true
Status:       Running
IP:           192.168.135.151
IPs:
  IP:           192.168.135.151
Controlled By:  ReplicaSet/nginx-ingress-controller-7bbb744996
Containers:
  nginx-ingress-controller:
    Container ID:  containerd://f0f3c6cb2aafb5bc7474fc44c7b680aede4a481ef0b0c8826a71af88ffa0e663
    Image:         registry.aliyuncs.com/kubernetes/nginx-ingress-controller:0.20.0
    Image ID:      registry.aliyuncs.com/kubernetes/nginx-ingress-controller@sha256:3f06079f7727b2fb7ad5c97d8152eb622ae504674395dfa71fda7ce315aaaf30
    Ports:         80/TCP, 443/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=$(POD_NAMESPACE)/default-http-backend
      --configmap=$(POD_NAMESPACE)/nginx-configuration
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --publish-service=$(POD_NAMESPACE)/ingress-nginx
      --annotations-prefix=nginx.ingress.kubernetes.io
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Sun, 15 Aug 2021 12:44:59 +0800
      Finished:     Sun, 15 Aug 2021 12:45:38 +0800
    Ready:          False
    Restart Count:  17
    Liveness:       http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-controller-7bbb744996-rd9d6 (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mj82d (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-mj82d:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  43m                    default-scheduler  Successfully assigned ingress-nginx/nginx-ingress-controller-7bbb744996-rd9d6 to ck8s2
  Normal   Created    42m (x2 over 43m)      kubelet            Created container nginx-ingress-controller
  Normal   Started    42m (x2 over 43m)      kubelet            Started container nginx-ingress-controller
  Normal   Pulled     41m (x3 over 43m)      kubelet            Container image "registry.aliyuncs.com/kubernetes/nginx-ingress-controller:0.20.0" already present on machine
  Warning  Unhealthy  41m (x10 over 43m)     kubelet            Readiness probe failed: Get "http://192.168.135.151:10254/healthz": dial tcp 192.168.135.151:10254: connect: connection refused
  Normal   Killing    41m (x2 over 42m)      kubelet            Container nginx-ingress-controller failed liveness probe, will be restarted
  Warning  Unhealthy  22m (x34 over 42m)     kubelet            Liveness probe failed: Get "http://192.168.135.151:10254/healthz": dial tcp 192.168.135.151:10254: connect: connection refused
  Warning  BackOff    2m56s (x147 over 40m)  kubelet            Back-off restarting failed container
[root@ck8s1 ingress]# 

查看pod日志

[root@ck8s1 ingress]# kubectl logs nginx-ingress-controller-7bbb744996-rd9d6 -n ingress-nginx
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.20.0
  Build:      git-e8d8103
  Repository: https://github.com/kubernetes/ingress-nginx.git
-------------------------------------------------------------------------------

nginx version: nginx/1.15.5
W0815 04:44:59.166129       8 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0815 04:44:59.166490       8 main.go:196] Creating API client for https://10.10.0.1:443

发现是443端口的问题，测试端口不通

[root@ck8s1 ingress]# telnet 10.0.0.1 443
Trying 10.0.0.1...

1，解决方法：
修改kube-proxy配置把流量转发该成ipvs模式

2，编辑配置：
kubectl edit configmap kube-proxy -n kube-system
将 mode: “” 为空 改成 mode: “ipvs”
将 masqueradeAll: null 改成 masqueradeAll: yes

3，删除所有kube-proxy的pod

 kubectl get pod -n kube-system | grep kube-proxy | awk '{system(" kubectl delete pod "$1" -n kube-system")}'

4，校验
kubectl logs kube-proxy-xxx -n kube-system 检查是否日志出现Using ipvs Proxier

5，重新部署ingress-nginx

[root@ck8s1 ingress]# kubectl get pod -n ingress-nginx                
NAME                                        READY   STATUS    RESTARTS   AGE
default-http-backend-7b6d9847f6-5jt5b       1/1     Running   0          13m
nginx-ingress-controller-7bbb744996-2c7tp   1/1     Running   0          13m

启动正常，没有报错