kubernetes：解决无法dns解析的问题

解决：

/ # nslookup myservice  不能解析的问题

问题所在：kube-dns错误

[kubeadm@server2 manifest]$ kubectl get pod -o wide
NAME                               READY   STATUS    RESTARTS   AGE   IP            NODE      NOMINATED NODE   READINESS GATES
nginx-deployment-b97c4ff7f-74sfg   1/1     Running   0          35h   10.244.1.27   server3              
nginx-deployment-b97c4ff7f-c7wct   1/1     Running   0          35h   10.244.2.26   server4              
test                               1/1     Running   3          37h   10.244.1.26   server3              

[kubeadm@server2 manifest]$ kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1              443/TCP   5d14h
myservice    ClusterIP   10.111.9.105           80/TCP    33h

[kubeadm@server2 manifest]$ kubectl describe svc myservice 
Name:              myservice
Namespace:         default
Labels:            
Annotations:       Selector:  app=nginx
Type:              ClusterIP
IP:                10.111.9.105
Port:                80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.1.27:80,10.244.2.26:80
Session Affinity:  None
Events:            

当你创建了一个service，你的node节点的kube-ipvs0上就会有这个service的ip

[root@server3 ~]# ip addr show
6: kube-ipvs0:  mtu 1500 qdisc noop state DOWN group default 
    link/ether 2a:e0:04:08:47:23 brd ff:ff:ff:ff:ff:ff
    inet 10.96.0.10/32 brd 10.96.0.10 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.96.0.1/32 brd 10.96.0.1 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.106.5.122/32 brd 10.106.5.122 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.111.9.105/32 brd 10.111.9.105 scope global kube-ipvs0
       valid_lft forever preferred_lft forever

查看策略

[root@server3 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.111.9.105:80 rr
  -> 10.244.1.27:80               Masq    1      0          0         
  -> 10.244.2.26:80               Masq    1      0          0   

查看dns解析：
master节点上对应的node地址为10.244.0.10、10.244.0.11

[kubeadm@server2 manifest]$ kubectl -n kube-system get pod -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP            NODE      NOMINATED NODE   READINESS GATES
coredns-7b8f97b6db-5g4hh          1/1     Running   4          5d14h   10.244.0.10   server2   <none>           <none>
coredns-7b8f97b6db-jxccd          1/1     Running   4          5d14h   10.244.0.11   server2   <none>           <none>

dns地址为10.96.0.10

[kubeadm@server2 manifest]$ kubectl -n kube-system get svc
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   5d14h

而节点上解析道的地址为：与master节点上对应的node地址10.244.0.10、10.244.0.11不同，这就是通过svc无法解析的原因，ipvs的方式没有同步更新策略

[root@server3 ~]# ipvsadm -ln
UDP  10.96.0.10:53 rr
  -> 10.244.0.6:53                Masq    1      0          0         
  -> 10.244.0.7:53                Masq    1      0          0         

如果修改解析，可以nslookup到

[kubeadm@server2 manifest]$ kubectl exec -it test -- sh
/ # vi /etc/resolv.conf 
/ # nslookup 10.244.0.10
Server:    10.244.0.10
Address 1: 10.244.0.10 10-244-0-10.kube-dns.kube-system.svc.cluster.local

Name:      10.244.0.10
Address 1: 10.244.0.10 10-244-0-10.kube-dns.kube-system.svc.cluster.local
/ # nslookup myservice.default.svc.cluster.local
Server:    10.244.0.10
Address 1: 10.244.0.10 10-244-0-10.kube-dns.kube-system.svc.cluster.local

Name:      myservice.default.svc.cluster.local
Address 1: 10.111.9.105 myservice.default.svc.cluster.local

查看日志会有大量报错

[kubeadm@server2 manifest]$ kubectl -n kube-system logs kube-proxy-24qlb

kube-dns服务的后端是正常的

[kubeadm@server2 manifest]$ kubectl -n kube-system describe svc kube-dns
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         10.244.0.10:53,10.244.0.11:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         10.244.0.10:53,10.244.0.11:53
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         10.244.0.10:9153,10.244.0.11:9153
Session Affinity:  None
Events:            <none>

问题原因所在：内核版本低了，需要将内核升级到4以上

[kubeadm@server2 manifest]$ uname -r
3.10.0-957.el7.x86_64

升级内核
最新内核下载地址：
https://mirrors.aliyun.com/centos/7.7.1908/virt/x86_64/xen-48/kernel-4.9.215-36.el7.x86_64.rpm

[root@server2 ~]# yum install -y kernel-4.9.215-36.el7.x86_64.rpm

[root@server2 ~]# cd /boot/grub2/
[root@server2 grub2]# ls
device.map  fonts  grub.cfg  grubenv  i386-pc  locale

[root@server2 grub2]# cat grub.cfg
Red Hat Enterprise Linux Server (4.9.215-36.el7.x86_64) 7.6 (Maipo)

[root@server2 grub2]# grub2-set-default 'Red Hat Enterprise Linux Server (4.9.215-36.el7.x86_64) 7.6 (Maipo)'

[root@server2 grub2]# reboot

[root@server2 ~]# uname -r  # 内核更新完成
4.9.215-36.el7.x86_64

master上的dnsip和nodeip

[kubeadm@server2 manifest]$ kubectl -n kube-system get svc
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   5d22h

[kubeadm@server2 manifest]$ kubectl get pod -n kube-system -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP            NODE      NOMINATED NODE   READINESS GATES
coredns-7b8f97b6db-5g4hh          1/1     Running   5          5d22h   10.244.0.12   server2   <none>           <none>
coredns-7b8f97b6db-jxccd          1/1     Running   5          5d22h   10.244.0.13   server2   <none>           <none>

查看节点上的ipvsadm策略是否与master上的ip一致

[root@server3 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn  
TCP  10.96.0.10:9153 rr  # 是一致的
  -> 10.244.0.12:9153             Masq    1      0          0         
  -> 10.244.0.13:9153             Masq    1      0          0   

创建pod

[kubeadm@server2 manifest]$ cat deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: myapp:v2
        ports:
        - containerPort: 80

[kubeadm@server2 manifest]$ kubectl apply -f deployment.yaml 
deployment.apps/nginx-deployment created

[kubeadm@server2 manifest]$ kubectl get pod -o wide
NAME                               READY   STATUS    RESTARTS   AGE   IP            NODE      NOMINATED NODE   READINESS GATES
nginx-deployment-b97c4ff7f-9zxxf   1/1     Running   0          11s   10.244.2.32   server4   <none>           <none>
nginx-deployment-b97c4ff7f-krq5z   1/1     Running   0          11s   10.244.1.35   server3   <none>           <none>

[kubeadm@server2 manifest]$ kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   5d22h
myservice    ClusterIP   10.111.9.105   <none>        80/TCP    41h

节点的ipvsadm策略加入了新建pod的ip

[root@server3 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.111.9.105:80 rr
  -> 10.244.1.35:80               Masq    1      0          0         
  -> 10.244.2.32:80               Masq    1      0          0 

测试解析：可以解析，问题解决

[kubeadm@server2 manifest]$ kubectl run test -it --image=busyboxplus
If you don't see a command prompt, try pressing enter.
/ # nslookup myservice
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      myservice
Address 1: 10.111.9.105 myservice.default.svc.cluster.local
/ # curl myservice
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>
/ # curl myservice/hostname.html
nginx-deployment-b97c4ff7f-9zxxf
/ # curl myservice/hostname.html
nginx-deployment-b97c4ff7f-krq5z