[Kubernetes]External-dns +CoreDns 安装
这里的前提是集群已安装Ingress，我这安装的是Ingress Nginx。
官方安装教程：https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/coredns.md
我们知道Ingress可以对集群中的服务进行DNS解析，意即我们可以通过类似http://nginx.demo.com/api 的方式访问集群中的服务，当然前提是，要在/etc/hosts中添加如下内容：
192.168.106.130 nginx.demo.com
external-dns 就是为了进一步简化这个问题，external-dns每隔一段时间会去查询Ingress，获得Ingress中的数据，然后push到其provider中，这里演示的是coredns。
必须说明一下，这里使用coredns演示，而coredns只能用于集群内部使用。external-dns+coredns的安装方法可参考官方的安装教程。官方教程说的不是很清楚的地方在etcd的使用。这里根据对etcd的使用有两种安装方式：
- 为external-dns新建一个etcd数据库
先看external-dns-etcd.yaml的内容：
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: coredns-etcd
name: coredns-etcd
namespace: default
spec:
ports:
- name: etcd-http
port: 2379
- name: etcd-peer
port: 2380
selector:
k8s-app: coredns-etcd
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: coredns-etcd
name: coredns-etcd
namespace: default
spec:
replicas: 1
selector:
matchLabels:
k8s-app: coredns-etcd
template:
metadata:
labels:
k8s-app: coredns-etcd
spec:
priorityClassName: system-cluster-critical
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/master
containers:
- name: coredns-etcd
image: k8s.gcr.io/etcd:3.5.1-0
command: ["/usr/local/bin/etcd"]
args:
- --listen-client-urls=http://0.0.0.0:2379
- --advertise-client-urls=http://0.0.0.0:2379
- --data-dir=/var/lib/etcd
livenessProbe:
failureThreshold: 5
tcpSocket:
port: 2379
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
ports:
- name: etcd-http
containerPort: 2379
protocol: TCP
- name: etcd-peer
containerPort: 2380
protocol: TCP
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: etcd-data
mountPath: /var/lib/etcd
volumes:
- name: etcd-data
hostPath:
path: /var/lib/coredns-etcd
type: DirectoryOrCreate
然后kubectl apply -f external-dns-etcd.yaml即可。
- 安装coredns
下载coredns的values.yaml文档：
wget https://raw.githubusercontent.com/helm/charts/HEAD/stable/coredns/values.yaml
修改：endpoint http://coredns-etcd:2379，其它部分安装官方文档提示修改。
接下来就可以使用helm安装coredns，可能需要更新helm repo：
helm repo add stable https://charts.helm.sh/stable
安装命令为：
helm install my-coredns stable/coredns --f values.yaml
2）安装external-dns
external-dns.yaml内容为：
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: external-dns
rules:
- apiGroups: [""]
resources: ["services","endpoints","pods"]
verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get","watch","list"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: external-dns-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: external-dns
subjects:
- kind: ServiceAccount
name: external-dns
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-dns
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns
namespace: kube-system
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: external-dns
template:
metadata:
labels:
app: external-dns
spec:
serviceAccountName: external-dns
containers:
- name: external-dns
image: bitnami/external-dns:0.11.1
args:
- --source=ingress
- --provider=coredns
- --log-level=debug # debug only
env:
- name: ETCD_URLS
value: "http://coredns-etcd:2379"
运行 kubectl apply -f external-dns.yaml即可，运行成功后，在集群任意机器上：
[root@master1 k8s-manual-files]# dig @10.101.12.230 nginx.example.org +short
192.168.106.120
注意前面的10.101.12.230是coredns service的IP:
[root@master1 k8s-manual-files]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-coredns-coredns ClusterIP 10.101.12.230 53/UDP,53/TCP
如果要在浏览器中直接访问http://nginx.example.org，需要修改vim /etc/resolv.conf：
# Generated by NetworkManager
search localdomain example.org
nameserver 192.168.106.2
nameserver 10.101.12.230
- 使用集群中已安装的etcd数据库
使用集群中的etcd与前面的差别是，集群中的etcd是没有提供service的，只提供了一个广播地址，例如https://192.168.106.135:2379，访问这个地址是需要etcd的签名文件的。
1）首先来看corefile的定义：
kubernetes [ZONES…] {
resyncperiod DURATION
endpoint URL [URL…]
tls CERT KEY CACERT
namespaces NAMESPACE…
labels EXPRESSION
pods POD-MODE
endpoint_pod_names
upstream [ADDRESS…]
ttl TTL
fallthrough [ZONES…]
}
tls： 用于指定连接远程kubernetes api（ETCD）的相关证书。示例：
tls admin.pem admin-key.pem ca.pem
这里与ETCD中的设置对比：
- –etcd-cafile=/etc/etcd/ssl/etcd-ca.pem
- –etcd-certfile=/etc/etcd/ssl/etcd.pem
- –etcd-keyfile=/etc/etcd/ssl/etcd-key.pem
然后tls的设置：
tls /etc/coredns/tls/etcd/etcd.pem /etc/coredns/tls/etcd/etcd-key.pem /etc/coredns/tls/etcd/etcd-ca.pem
2）修改配置文件
为了在coredns和external-dns中使用etcd的证书，首先必须在集群中生成一个secret：
kubectl create secret generic etcd-client-certs --from-file=etcd-ca.pem=/etc/etcd/ssl/etcd-ca.pem --from-file=etcd.pem=/etc/etcd/ssl/etcd.pem --from-file=etcd-key.pem=/etc/etcd/ssl/etcd-key.pem
在coredns的values中：
- name: etcd
parameters: example.org
configBlock: |-
stubzones
path /skydns
endpoint https://192.168.106.135:2379
tls /etc/coredns/tls/etcd/etcd.pem /etc/coredns/tls/etcd/etcd-key.pem /etc/coredns/tls/etcd/etcd-ca.pem
extraSecrets:
- name: etcd-client-certs
mountPath: /etc/coredns/tls/etcd
# - name: some-fancy-secret
# mountPath: /etc/wherever
在external-dns.yaml中;
containers:
- name: external-dns
image: bitnami/external-dns:0.11.1
args:
- --source=ingress
- --provider=coredns
- --log-level=debug # debug only
env:
- name: ETCD_URLS
value: "https://192.168.106.135:2379"
- name: ETCD_CERT_FILE
value: "/etc/coredns/tls/etcd/etcd.pem"
- name: ETCD_KEY_FILE
value: "/etc/coredns/tls/etcd/etcd-key.pem"
- name: ETCD_CA_FILE
value: "/etc/coredns/tls/etcd/etcd-ca.pem"
volumeMounts:
- mountPath: /etc/coredns/tls/etcd
name: etcd-client-certs
volumes:
- secret:
defaultMode: 420
secretName: etcd-client-certs
name: etcd-client-certs
二者的安装方式同上，安装完成后，dig @10.101.12.230 nginx.example.org +short 会得到同样结果。
当然在这里external-dns也可以使用helm安装，其values：
https://github.com/helm/charts/blob/master/stable/external-dns/values.yaml
https://github.com/helm/charts/tree/master/stable/external-dns
然后安装方式：
helm install my-release bitnami/external-dns -f values.yaml
部分修改后的配置：
coredns:
## Comma-separated list of the etcd endpoints
## Secure (https) endpoints can be used as well, in that case `etcdTLS` section
## should be filled in accordingly
##
etcdEndpoints: "https://192.168.106.135:2379"
## Configuration of the secure communication and client authentication to the etcd cluster
## If enabled all the values under this key must hold a valid data
##
etcdTLS:
## Enable or disable secure communication and client authentication to the etcd cluster
##
enabled: true
## Name of the existing secret containing cert files for client communication
## ref: https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/security.md
## ref (secret creation):
## https://github.com/bitnami/charts/tree/master/bitnami/etcd#configure-certificates-for-client-communication
##
secretName: "etcd-client-certs"
## Location of the mounted certificates inside external-dns pod
##
mountPath: "/etc/coredns/tls/etcd"
## CA PEM file used to sign etcd TLS cert, should exist in the secret provided above
##
caFilename: "etcd-ca.pem"
## Certificate PEM file, should exist in the secret provided above
## Will be used by external-dns to authenticate against etcd
##
certFilename: "etcd.pem"
## Private key PEM file, should exist in the secret provided above
## Will be used by external-dns to authenticate against etcd
##
keyFilename: "etcd-key.pem"
- 问题
1）使用最新版的coredns/coredns，用helm安装后有如下问题：
W0610 08:47:25.183137 1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User “system:serviceaccount:default:default” cannot list resource “endpointslices” in API group “discovery.k8s.io” at the cluster scope
E0610 08:47:25.183282 1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User “system:serviceaccount:default:default” cannot list resource “endpointslices” in API group “discovery.k8s.io” at the cluster scope
这需要修改cluserrole的内容，首先通过kubectl get clusterrole找到相关的role：
[root@master1 k8s-manual-files]# kubectl get clusterrole
NAME CREATED AT
admin 2022-05-27T06:12:52Z
my-coredns-coredns 2022-06-10T08:45:24Z
通过kubectl describe clusterrole my-coredns-coredns查看：
kubectl describe clusterrole my-coredns-coredns
Name: my-coredns-coredns
Labels: app.kubernetes.io/instance=my-coredns
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=coredns
helm.sh/chart=coredns-1.13.8
Annotations: meta.helm.sh/release-name: my-coredns
meta.helm.sh/release-namespace: default
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
endpoints [] [] [list watch]
namespaces [] [] [list watch]
pods [] [] [list watch]
services [] [] [list watch]
然后，通过kubectl edit clusterrole my-coredns-coredns修改：
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
meta.helm.sh/release-name: my-coredns
meta.helm.sh/release-namespace: default
creationTimestamp: "2022-06-10T08:45:24Z"
labels:
app.kubernetes.io/instance: my-coredns
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: coredns
helm.sh/chart: coredns-1.13.8
name: my-coredns-coredns
resourceVersion: "704504"
uid: ea966c89-206c-490a-aa47-f0c54e30cebe
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
添加的内容为：
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
保存后，等待一会儿，运行kubectl get po就会发现coredns已ok。
2）WARNING: recursion requested but not available
这个问题暂时不知道怎么解决，当/etc/resolv.conf存在多个nameserver时，运行nslookup时会出现：Got recursion not available from … trying next server 这个问题，删除原有的内容，改为：
# Generated by NetworkManager
search examole.org
nameserver 10.99.69.85
然后运行就正常了：
[root@master1 k8s-manual-files]# nslookup nginx.example.org
Server: 10.99.69.85
Address: 10.99.69.85#53
Name: nginx.example.org
Address: 192.168.106.120
在浏览器输入nginx.example.org也可以看到正确的内容了。