一,理解Kubedns原理
通俗理解,首先明白k8s dns是为服务的发现而生,即service的发现,为了能够让其它服务能够直接通过service 名字找到它们,于是就需要dns将service名转换为它的VIP, 那么service的变化如何知道? 如何知道目前有哪些service 及知道他们的vip呢,service的增加减少又如何知道了,
所以要有一个组件(1.3中是Kube2sky,1.3后是kubedns)来时刻监控它的变化,监控到了什么就把它记录下来,记录的是service和ip之间的映射关系,称为DNS解析记录,但是记录在某个位置呢,监控的组件不同,记录的位置也不同, Kube2sky是记录到etcd中,kubedns是记录到哪了呢, 记录在内存当中,使用树形结构在内存中保存监控到的记录。
由于一切对集群的操作都是通过其API来的,所以它监控service资源变化也是借助于k8s API
这个组件仅仅是记录下来以后,并不提供查询。所以查询就是另外一个组件的作用了,1.3之前是由Skydns来查询的, 所有服务(pod)都找skydns查询,1.3以后都是由dnsmasq查询的,它们从前面记录的地方查询相关service的解析记录,之所以用dnsmasq来查询,因为它提供DNS查询缓存,在内存中完成查询,查询速度非常快!
两个组件一个负责监控并记录,另外一个负责帮其它服务做查询, 两个组件必须要同时运行,如果其中一个出问题了,那么dns系统将无法解析,于是需要有另一个组件专门来监控它们的健康状态,这个组件就是exechealthz,Exechealthz是两个版本中唯一保留的容器,依然提供健康检查。
在K8s中,这三个组件都是用容器运行的,而且是在同一个pod中,
官方理解,上图:
● dnsmasq简介
Dnsmasq是一款小巧的DNS配置工具
在kube-dns插件中的作用:
● 通过kubedns容器获取DNS规则,在集群中提供DNS查询服务,相当于dns的server端。
● 提供DNS缓存,提高查询性能
● 降低kubedns容器的压力、提高稳定性
● Dockerfile在GitHub上Kubernetes组织的contrib仓库中,位于dnsmasq目录下
● exechealthz简介
● 在kube-dns插件中提供健康检查功能
● 源码同样在contrib仓库中,位于exec-healthz目录下。
● 新版中会对两个容器都进行健康检查,更加完善。
● 总结
kube-dns插件的三个容器的功能如下:
● kubedns容器
● 监视k8s Service资源并更新DNS记录
● 替换etcd,使用TreeCache数据结构保存DNS记录并实现SkyDNS的Backend接口
● 接入SkyDNS,对dnsmasq提供DNS查询服务
● dnsmasq容器
● 对集群提供DNS查询服务
● 设置kubedns为upstream
● 提供DNS缓存,降低kubedns负载,提高性能
● exechealthz容器
● 定期检查kubedns和dnsmasq的健康状态
● 为k8s活性检测提供HTTP API
二、部署kubedns
1,上官方网址下载需要的yaml部署文件:https://github.com/kubernetes/kubernetes/tree/release-1.8/cluster/addons/dns
$ ls *.yaml *.base kubedns-cm.yaml kubedns-sa.yaml kubedns-controller.yaml.base kubedns-svc.yaml.base
2,主要是修改这四个部署文件
cat kubedns-cm.yaml #此文件不需修改 apiVersion: v1 kind: ConfigMap metadata: name: kube-dns namespace: kube-system labels: addonmanager.kubernetes.io/mode: EnsureExists
# cat kubedns-sa.yaml #此文件不需修改 apiVersion: v1 kind: ServiceAccount metadata: name: kube-dns namespace: kube-system labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile
接下来主要修改的是 kubedns-controller.yaml 和 kubedns-svc.yaml
主要diff来比较下原文件与修改过的文件区别:
diff kubedns-controller.yaml.base /root/kubernetes/k8s-deploy/mainifest/dns/kubedns-controller.yaml 58c58 < image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5 --- > image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4 88c88 < - --domain=$DNS_DOMAIN. --- > - --domain=cluster.local. 109c109 < image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5 --- > image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4 128c128 < - --server=/$DNS_DOMAIN/127.0.0.1#10053 --- > - --server=/cluster.local/127.0.0.1#10053 147c147 < image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5 --- > image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4 160,161c160,161 < - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.$DNS_DOMAIN,5,A < - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.$DNS_DOMAIN,5,A --- > - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A > - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A 无非就是将有 k8s-dns-sidecar-amd64:1.14.5 的改为 k8s-dns-sidecar-amd64:1.14.4,将变量集群域名的环境变量 $DNS_DOMAIN 替换我们之前设置好的域名,如cluster.local。
修改过后的完整文件内容如下:
# cat kubedns-controller.yaml # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Should keep target in cluster/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml # in sync with this file. # __MACHINE_GENERATED_WARNING__ apiVersion: extensions/v1beta1 kind: Deployment metadata: name: kube-dns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: # replicas: not specified here: # 1. In order to make Addon Manager do not reconcile this replicas parameter. # 2. Default is 1. # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on. strategy: rollingUpdate: maxSurge: 10% maxUnavailable: 0 selector: matchLabels: k8s-app: kube-dns template: metadata: labels: k8s-app: kube-dns annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: tolerations: - key: "CriticalAddonsOnly" operator: "Exists" volumes: - name: kube-dns-config configMap: name: kube-dns optional: true containers: - name: kubedns image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4 resources: # TODO: Set memory limits when we've profiled the container for large # clusters, then set request = limit to keep this container in # guaranteed class. Currently, this container falls into the # "burstable" category so the kubelet doesn't backoff from restarting it. limits: memory: 170Mi requests: cpu: 100m memory: 70Mi livenessProbe: httpGet: path: /healthcheck/kubedns port: 10054 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 readinessProbe: httpGet: path: /readiness port: 8081 scheme: HTTP # we poll on pod startup for the Kubernetes master service and # only setup the /readiness HTTP server once that's available. initialDelaySeconds: 3 timeoutSeconds: 5 args: - --domain=cluster.local. - --dns-port=10053 - --config-dir=/kube-dns-config - --v=2 env: - name: PROMETHEUS_PORT value: "10055" ports: - containerPort: 10053 name: dns-local protocol: UDP - containerPort: 10053 name: dns-tcp-local protocol: TCP - containerPort: 10055 name: metrics protocol: TCP volumeMounts: - name: kube-dns-config mountPath: /kube-dns-config - name: dnsmasq image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4 livenessProbe: httpGet: path: /healthcheck/dnsmasq port: 10054 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 args: - -v=2 - -logtostderr - -configDir=/etc/k8s/dns/dnsmasq-nanny - -restartDnsmasq=true - -- - -k - --cache-size=1000 - --log-facility=- - --server=/cluster.local/127.0.0.1#10053 - --server=/in-addr.arpa/127.0.0.1#10053 - --server=/ip6.arpa/127.0.0.1#10053 ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP # see: https://github.com/kubernetes/kubernetes/issues/29055 for details resources: requests: cpu: 150m memory: 20Mi volumeMounts: - name: kube-dns-config mountPath: /etc/k8s/dns/dnsmasq-nanny - name: sidecar image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4 livenessProbe: httpGet: path: /metrics port: 10054 scheme: HTTP initialDelaySeconds: 60 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 args: - --v=2 - --logtostderr - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A ports: - containerPort: 10054 name: metrics protocol: TCP resources: requests: memory: 20Mi cpu: 10m dnsPolicy: Default # Don't use cluster DNS. serviceAccountName: kube-dns
kubedns-controller 运行了 三个容器:kubedns dnsmasq sidecar,sidecar 是一个监控健康模块,同时向外暴露metrics 记录。
cat kubedns-svc.yaml # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # __MACHINE_GENERATED_WARNING__ apiVersion: v1 kind: Service metadata: name: kube-dns namespace: kube-system labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "KubeDNS" spec: selector: k8s-app: kube-dns clusterIP: 10.254.0.2 #修改为我们设定的cluster的IP,其它默认。 ports: - name: dns port: 53 protocol: UDP - name: dns-tcp port: 53 protocol: TCP
部署文件设置好后,一起来创建:
# kubectl create -f kubedns-cm.yaml -f kubedns-sa.yaml -f kubedns-controller.yaml -f kubedns-svc.yaml
查看创建后的pod,svc:
# kubectl get pod,svc -n kube-system NAME READY STATUS RESTARTS AGE po/kube-dns-7797cb8758-dlhqs 3/3 Running 0 3h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE svc/kube-dns ClusterIP 10.254.0.253/UDP,53/TCP 3h
三,验证kubedns功能
如果我们此前已经有了部署了许多pod和服务,并且在Kubelet 启动配置文件加入了 --cluster-dns=10.254.0.2 --cluster-domain=cluster.local 参数,则可以进入某个pod中的容器内查看其/etc/resolv.conf 文件
root@jekins:~# cat /etc/resolv.conf nameserver 10.254.0.2 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5 如上所示,根据kubelet的启动参数,kubelet会在每个pod中设置DNS域名解析文件/etc/resolv.conf,加入了nameser和search搜索域。最后应用程序就能够像访问网站一样,仅仅通过服务的名字就能访问到服务了。 root@jek:~# ping jenkinsservice PING jenkinsservice.default.svc.cluster.local (10.254.145.97) 56(84) bytes of data. 可以看到已经解析出来了
通常可以启动一个带有nslookup工具的pod,比如busybox,镜像从gcr.io/google_containers/busybox下载,来验证DNS服务是否能够正常工作。
四,kubedns自动水平扩展
在上面部署的Kubedns,它只有一个pod,如果这个pod没有了 又或一个pod因为解析量大无法撑起解析时,会导致dns无法工作,所以官方给了一个自动水平伸缩的方案,根据当前pod的负载来决定是否增加或减少pod数量,
称为,dns-horizontal-autoscaler ,部署文件位于kubernetes/cluster/addons/dns-horizontal-autoscale,下载下来可直接部署。
$ /data/k8sdns# ls dns-horizontal-autoscaler* dns-horizontal-autoscaler-rbac.yaml dns-horizontal-autoscaler.yaml
dns-horizontal-autoscaler-rbac.yaml文件解析:
实际它就创建了三个资源:ServiceAccount、ClusterRole、ClusterRoleBinding ,创建帐户,创建角色,赋予权限,将帐户绑定到角色上面。
而dns-horizontal-autoscaler.yaml 文件它就创建一个Deployment资源,里面跑了一个autoscaler的容器,直接部署即可。
现在来部署: 先部署RBAC文件,再部署服务文件:
# kubectl create -f dns-horizontal-autoscaler-rbac.yaml serviceaccount "kube-dns-autoscaler" created clusterrole "system:kube-dns-autoscaler" created clusterrolebinding "system:kube-dns-autoscaler" created # kubectl create -f dns-horizontal-autoscaler.yaml deployment "kube-dns-autoscaler" created
查看创建状态 ,并且可以看到在创建之前dns只有一个pod:
# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE kube-dns-7797cb8758-dlhqs 3/3 Running 3 19h kube-dns-autoscaler-7db47cb9b7-gq5wk 0/1 ContainerCreating 0 21s
过几分钟后查看,就会自动创建出来一个dns pod:
# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE kube-dns-7797cb8758-dlhqs 3/3 Running 3 19h kube-dns-7797cb8758-kwmqr 0/3 ContainerCreating 0 48s kube-dns-autoscaler-7db47cb9b7-gq5wk 1/1 Running 0 1m
至此,一个完整的kubedns部署完成了。