k8s集群部署DNS服务发现
环境：
- os: centos
- kubernetes: 1.5.2
- docker: 1.12.5
一、skydns-rc.yaml 文件
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-dns-v9
namespace: default
labels:
k8s-app: kube-dns
version: v9
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-dns
version: v9
template:
metadata:
labels:
k8s-app: kube-dns
version: v9
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: etcd
image: 192.168.100.90:5000/duni/etcd-amd64:3.0.17
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 100m
memory: 50Mi
command:
- /usr/local/bin/etcd
# - --privileged=true
- -data-dir
- /home/data/etcd
- -listen-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -advertise-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -initial-cluster-token
- skydns-etcd
volumeMounts:
- mountPath: /home/data/etcd
name: etcd-storage
- name: kube2sky
#image: gcr.io/google_containers/kube2sky:1.11
image: 192.168.100.90:5000/duni/kube2sky:1.14
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 100m
memory: 50Mi
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /readiness
port: 8081
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
args:
# command = "/kube2sky"
# - -etcd-server=http://127.0.0.1:4001
#- -kube_master_url=http://172.27.8.210:8080
- --kube-master-url=http://192.168.100.27:8080
- --domain=cluster.local
- name: skydns
#image: gcr.io/google_containers/skydns:2015-03-11-001
image: 192.168.100.90:5000/duni/skydns:2015-10-13-8c72f8c
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 100m
memory: 50Mi
args:
# command = "/skydns"
- -machines=http://127.0.0.1:2379
- -addr=0.0.0.0:53
- -ns-rotate=false
- -domain=cluster.local
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 1
timeoutSeconds: 5
- name: healthz
#image: gcr.io/google_containers/exechealthz:1.0
image: 192.168.100.90:5000/duni/exechealthz-amd64:latest
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 10m
memory: 20Mi
args:
- -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
- -port=8080
ports:
- containerPort: 8080
protocol: TCP
volumes:
- name: etcd-storage
emptyDir: {}
dnsPolicy: Default # Don't use cluster DNS.
skydns-rc.yaml文件中需要根据自己的实际情况进行相关修改
1、把镜像改为自己私有仓库路径
$ cat skydns-rc.yaml | grep image
image: 192.168.100.90:5000/duni/etcd-amd64:3.0.17
imagePullPolicy: IfNotPresent
#image: gcr.io/google_containers/kube2sky:1.11
image: 192.168.100.90:5000/duni/kube2sky:1.14
imagePullPolicy: IfNotPresent
#image: gcr.io/google_containers/skydns:2015-03-11-001
image: 192.168.100.90:5000/duni/skydns:2015-10-13-8c72f8c
imagePullPolicy: IfNotPresent
#image: gcr.io/google_containers/exechealthz:1.0
image: 192.168.100.90:5000/duni/exechealthz-amd64:latest
imagePullPolicy: IfNotPresent
建议镜像到阿里云容器镜像中查找，然后更改镜像tag，再push到自己搭建的私有仓库中，如何搭建自己的私有仓库
2、kube2sky容器中参数：- --kube-master-url=http://192.168.100.27:8080设为你自己的k8s集群master主机ip:port, - --domain=cluster.local 设置集群中service域名（可自行定义一个名字）
3、skydns容器中参数：- -domain=cluster.local，需跟kube2sky中设置的名称一致
二、skydns-svc.yaml 文件
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: default
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.254.0.100
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
此处可自行修改clusterIP: 10.254.0.100 ip
三、修改k8s集群中节点机的kubelet启动参数
$ vi /etc/kubernetes/kubelet # centos系统，ubuntu系统在不同路径下
KUBELET_ARGS="--cluster_dns=10.254.0.100 --cluster_domain=cluster.local"
注：--cluster_dns必须跟skydns-svc.yaml文件中的clusterIP值相同，cluster_domain必须跟skydns-rc.yaml文件中skydns和kube2sky容器中的domain参数值相同
重启kubelet
systemctl restart kubelet
四、创建dns pod和service
kubectl create -f skydns-rc.yaml
kubectl create -f skydns-svc.yaml
查看pod和service状态
$ kubectl get pods --all-namespaces | grep kube-dns
NAMESPACE NAME READY STATUS RESTARTS AGE
default kube-dns-v9-vldgj 4/4 Running 0 3h
$ kubectl get pods --all-namespaces | grep kube-dns
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kube-dns 10.254.0.100 53/UDP,53/TCP 3h
五、验证dns
busybox.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- image: busybox
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: busybox
restartPolicy: Always
创建busybox pod
$ kubectl create -f busybox.yaml
$ kubectl get pods --all-namespaces | grep busybox
NAMESPACE NAME READY STATUS RESTARTS AGE
default busybox 1/1 Running 3 3h
验证dns解析
# 查看所有的service（以下是我master主机所有服务）
$ kbuectl get svc --all-namespaces
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default frontend 10.254.204.132 80:30001/TCP 19h
default kube-dns 10.254.0.100 53/UDP,53/TCP 3h
default kubernetes 10.254.0.1 443/TCP 30d
default mysql-service 10.254.246.96 3306:30365/TCP 21h
default redis-master 10.254.172.30 6379/TCP 19h
default redis-service 10.254.253.213 6379/TCP 19h
kube-system heapster 10.254.145.32 80/TCP 18h
kube-system kubernetes-dashboard 10.254.163.216 80:30009/TCP 20h
kube-system monitoring-grafana 10.254.199.203 80/TCP 18h
kube-system monitoring-influxdb 10.254.27.3 8086/TCP 18h
# 验证
$ kubectl exec -it busybox nslookup kubernetes
Server: 10.254.0.100
Address 1: 10.254.0.100
nslookup: can't resolve 'kubernetes'
$ kubectl exec -it busybox nslookup kubernetes.default
Server: 10.254.0.100
Address 1: 10.254.0.100
nslookup: can't resolve 'kubernetes.default': Try again
$ kubectl exec -it busybox nslookup www.baidu.com
Server: 10.254.0.100
Address 1: 10.254.0.100
Name: www.baidu.com
Address 1: 14.215.177.38
Address 2: 14.215.177.37
好吧，所有pod运行正常，可dns就是不能根据本地域名解析其ip，发现解析百度，网易等却可以
六、查找dns不能根据域名解析其ip的原因
查看dns pod中各个容器的日志，查看skydns容器时，发现以下错误，请求超时
$ kubectl logs --namespace=default $(kubectl get pods --namespace=default -l k8s-app=kube-dns -o name) -c skydns
2017-04-26T07:15:35.141855000Z 2017/04/26 07:15:35 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
2017-04-26T07:18:09.141845000Z 2017/04/26 07:18:09 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
2017-04-26T07:21:53.045513000Z 2017/04/26 07:21:53 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
2017-04-26T07:26:13.142510000Z 2017/04/26 07:26:13 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"
2017-04-26T07:28:25.045739000Z 2017/04/26 07:28:25 skydns: failure to forward request "read udp 192.168.100.1:53: i/o timeout"于是谷歌，找到原因所在，192.168.100.1 这个dns服务地址并不可用，那我们就用谷歌公开的dns
修改集群master主机，新增谷歌dns
$ vi /etc/resolv.conf
# Generated by NetworkManager
nameserver 192.168.100.1
# 新增下面两行
nameserver 8.8.8.8
nameserver 8.8.4.4
修改skydns-rc.yaml 中 skydns 容器的args参数
args:
# command = "/skydns"
- -machines=http://127.0.0.1:2379
- -addr=0.0.0.0:53
- -ns-rotate=false
- -domain=cluster.local
- -nameservers=8.8.8.8:53,8.8.4.4:53 # 新增这行
重启启动dns 和 busybox pod
$ kubectl delete rc kube-dns --namespace=default
$ kubectl delete src kube-dns --namespace=default
$ kubectl delete pods busybox --namespace=default
$ kubectl create -f skydns-rc.yaml
$ kubectl create -f skydns-svc.yaml
$ kubectl create -f busybox.yaml
七、再次验证dns
$ kubectl exec -it busybox nslookup kubernetes
Server: 10.254.0.100
Address 1: 10.254.0.100
Name: kubernetes
Address 1: 10.254.0.1
$ kubectl exec -it busybox nslookup kubernetes.default
Server: 10.254.0.100
Address 1: 10.254.0.100
Name: kubernetes.default
Address 1: 10.254.0.1
$ kubectl exec -it busybox nslookup heapster
Server: 10.254.0.100
Address 1: 10.254.0.100
nslookup: can't resolve 'heapster'
$ kubectl exec -it busybox nslookup heapster.kube-system
Server: 10.254.0.100
Address 1: 10.254.0.100
Name: heapster.kube-system
Address 1: 10.254.145.32
注：我们的dns pod所在的命名空间（namespace）是在default，当我们需要查询的服务跟dns pod不在同一命名空间时，需通过域名.命名空间 方式进行查找，如上面的heapster.kube-system
友情提示：一定要学会查看pod中的容器日志，因为很多时候，我们一个pod中某个容器只要一个参数书写错误，则该容器就running失败了
kubectl logs –namespace=namespace_name $(kubectl get pods –namespace=namespace_name -l label_name=label_value -o name) -c container_name
如上面查找dns pod中skydns容器日志
kubectl logs –namespace=default $(kubectl get pods –namespace=default -l k8s-app=kube-dns -o name) -c skydns