在prometheus operator中自定义监控项etcd

添加一个自定义监控的步骤：
第一步建立一个 ServiceMonitor 对象，用于 Prometheus 添加监控项
第二步为 ServiceMonitor 对象关联 metrics 数据接口的一个 Service 对象
第三步确保 Service 对象可以正确获取到 metrics 数据

查看证书
$ kubectl get pods -n kube-system
etcd-master 1/1 Running 0 2h
$ kubectl get pod etcd-master -n kube-system -o yaml
ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key
使用上述3个证书创建secret
$ kubectl -n monitoring create secret generic etcd-certs --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key --from-file=/etc/kubernetes/pki/etcd/ca.crt
使用secret
$ kubectl get prometheus -n monitoring
$ kubectl edit prometheus k8s -n monitoring

replicas: 2
secrets:
- etcd-certs

$ kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring
/ $ ls /etc/prometheus/secrets/etcd-certs/
ca.crt healthcheck-client.crt healthcheck-client.key

创建ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd-k8s
  namespace: monitoring
  labels:
    k8s-app: etcd-k8s
spec:
  jobLabel: k8s-app
  endpoints:
  - port: port
    interval: 30s
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
      certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt
      keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key
      insecureSkipVerify: true
  selector:
    matchLabels:
      k8s-app: etcd
  namespaceSelector:
    matchNames:
    - kube-system

创建service

apiVersion: v1
kind: Service
metadata:
  name: etcd-k8s
  namespace: kube-system
  labels:
    k8s-app: etcd
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: port
    port: 2379
    protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
  name: etcd-k8s
  namespace: kube-system
  labels:
    k8s-app: etcd
subsets:
- addresses:
  - ip: 192.168.1.243
    nodeName: etc-master
  ports:
  - name: port
    port: 2379
    protocol: TCP

这里创建的 Service 没有采用通过 label 标签的形式去匹配 Pod 的做法，因为很多时候我们创建的 etcd 集群是独立于集群之外的，这种情况下面我们就需要自定义一个 Endpoints，Service 的 clusterIP 设置为 None
192.168.1.243是etcd所在的master地址

修改etcd的yaml文件
$ vim /etc/kubernetes/manifests/etcd.yaml
listen-client-urls=https://0.0.0.0:2379,https://192.168.1.243:2379