OpenShift在安装的时候依赖于NetworkManager服务完成dnsmasq自动配置,请确保该服务保护启动状态,并保证在网卡配置文件中不要添加 NM_CONTROLLED=no参数,否则会导致一系列网络问题。如果一定需要开启此参数,请做好心里准备,你要知道整个dns访问配置流程,需要手动维护下列配置文件:
/etc/resolv.conf
/etc/dnsmasq.d/origin-dns.conf
/etc/dnsmasq.d/origin-upstream-dns.conf
/etc/origin/node/resolv.conf
Pod访问外网dns查询流程:pod – dnsmasq – 上游dns(多个地址倒序查询)
Pod访问集群内服务dns查询流程:pod – dnsmasq – skydns(127.0.0.1:53) – 如果没有本地缓存 – openshift api
SkyDns:用于解析service域名 ,调用 OpenShift API 服务来获取主机名、IP地址等信息,然后封装成标准 DNS 记录并返回给查询客户端。
在node节点查看dns信息,可以看到dnsmasq启了3个进程,openshift启动了1个进程为skydns
#netstat -atunlp|grep 53
tcp 0 0 10.10.6.136:53 0.0.0.0:* LISTEN 22595/dnsmasq
tcp 0 0 172.17.0.1:53 0.0.0.0:* LISTEN 22595/dnsmasq
tcp 0 0 10.131.0.1:53 0.0.0.0:* LISTEN 22595/dnsmasq
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 8542/openshift
其中通过dnsmasq启动的53端口监听的网络与route -n查看到的网段对应
10.10.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
10.128.0.0 0.0.0.0 255.252.0.0 U 0 0 0 tun0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.30.0.0 0.0.0.0 255.255.0.0 U 0 0 0 tun0
在node节点查看skydns信息,通过openshift进程启动。
#netstat -atunlp|grep 53
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 8542/openshift
node节点上的skydns从缓存或API获取数据返回客户端查询
所有节点都安装SkyDns,组成一个分布式集群,其中master节点额外启动8053端口
Pod DNS:
#cat /etc/resolv.conf
nameserver 10.10.6.136
search nginx-10-10-163-88.svc.cluster.local svc.cluster.local cluster.local localdomain
options ndots:5
注解:
nameserver是宿主机ip地址,所有dns请求会发送到宿主机53端口
宿主机DNS:
#cat /etc/resolv.conf
#nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
#Generated by NetworkManager
search cluster.local
nameserver 10.10.6.136
安装启动NetworkManager会触发/etc/NetworkManager/dispatcher.d/99-origin-dns.sh脚本,修改 /etc/resolv.conf、设置搜索域,以及将宿主机的默认 IP 作为 nameserver、创建 /etc/origin/node/resolv.conf、生成/etc/dnsmasq.d/origin-dns.conf,/etc/dnsmasq.d/origin-upstream-dns.conf两个配置文件、启动dnsmasq服务,其中origin-upstream-dns.conf 定义了上游DNS服务器,该地址通过DHCP获取,我这里没有生成成功,手动配置并重启dnsmasq
#cat origin-dns.conf
no-resolv
domain-needed
no-negcache
max-cache-ttl=1
enable-dbus
dns-forward-max=10000
cache-size=10000
bind-dynamic
min-port=1024
except-interface=lo
#End of config
#cat origin-upstream-dns.conf
server=10.10.255.1
server=10.10.255.2
server=10.10.153.132
#systemctl restart dnsmasq
dnsmasq查询顺序是自下而上,即先查询10.10.153.132
vi /etc/dnsmasq.conf
#For debugging purposes, log each DNS query as it passes through
#dnsmasq.
log-queries
#systemctl restart dnsmasq
dnsmasq查看日志
#journalctl -f -u dnsmasq
Jan 14 11:10:51 openshift2 dnsmasq[14678]: setting upstream servers from DBus
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 10.10.255.2#53
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 10.10.255.1#53
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 10.10.153.132#53
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
Jan 14 11:10:51 openshift2 dnsmasq[14678]: using nameserver 127.0.0.1#53 for domain cluster.local
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[AAAA] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[A] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: reply hawkular-metrics.openshift-infra.svc.cluster.local is 172.30.87.204
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[AAAA] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[A] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: cached hawkular-metrics.openshift-infra.svc.cluster.local is 172.30.87.204
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[AAAA] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
Jan 14 11:11:05 openshift2 dnsmasq[14678]: forwarded hawkular-metrics.openshift-infra.svc.cluster.local to 127.0.0.1
Jan 14 11:11:05 openshift2 dnsmasq[14678]: query[A] hawkular-metrics.openshift-infra.svc.cluster.local from 10.131.0.4
根据日志得知,dnsmasq把请求都转发到127.0.0.1:53上的skydns中(封装在openshift进程)node节点上的skydns从缓存或API获取数据返回客户端查询,可以看到示例中是走的cached
openshift中dns配置信息:
master节点
vim /etc/origin/master/master-config.yaml
dnsConfig:
bindAddress: 0.0.0.0:8053
bindNetwork: tcp4
node节点
vim /etc/origin/node/node-config.yaml
dnsBindAddress: 127.0.0.1:53
dnsDomain: cluster.local
dnsIP: 0.0.0.0
dnsNameservers: null
dnsRecursiveResolvConf: /etc/origin/node/resolv.conf
无论是在宿主机还是在pod内,均可以访问service,因为首先都要经过宿主机dns服务器127.0.0.1:53
#ping hawkular-metrics.openshift-infra.svc.cluster.local
PING hawkular-metrics.openshift-infra.svc.cluster.local (172.30.87.204) 56(84) bytes of data.
参考文档:
http://www.cnblogs.com/sammyliu/p/10056035.html
https://www.cnblogs.com/ericnie/p/10216775.html
https://blog.cloudtechgroup.cn/Blog/2018/07/23/ocp-2018-07-23/