Prometheus：监控与告警：15: 获取Kubernetes内置的cAdvisor进行监控

前面一篇文章中介绍了Kubernetes和Prometheus进行集成的常见方式，这篇文章结合具体的示例介绍一下如何将Kubernetes内置的cAdvisor作为Prometheus监控的数据来源。

集成cAdvisor

集成方式

Kubernetes主要提供了如下5种服务发现模式和Prometheus进行集成：

• Node
• Pod
• Endpoints
• Service
• Ingress

使用cAdvisor主要需要使用Node服务发现模式，配置方式如下所示

Node服务发现模式

        kubernetes_sd_configs:
        - role: node

监控方法

监控对象	监控指标内容	服务发现模式	监控方式	数据来源
集群各节点Kubelet组件	各节点Kubelet的基本运行状态相关的监控指标	node	白盒监控	Kubelet

Kubernetes环境准备

本文使用Kubernetes 1.17，可参看下文进行快速环境搭建：

• 单机版本或者集群版本环境搭建

[root@host131 ~]# kubectl get nodes -o wide
NAME              STATUS   ROLES    AGE    VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
192.168.163.131   Ready       116m   v1.17.0   192.168.163.131           CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://18.9.7
[root@host131 ~]# 

RBAC配置文件

准备如下RBAC配置文件

[root@host131 cadvisor]# cat rbac.yml 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
[root@host131 cadvisor]# 

ConfigMap设定文件

ConfigMap设定文件如下所示：

[root@host131 cadvisor]# cat configmap.yml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-configmap
  labels:
    name: prometheus-configmap
  namespace: default
data:
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s

    scrape_configs:
      - job_name: 'k8s-cadvisor'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        kubernetes_sd_configs:
        - role: node

        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
[root@host131 cadvisor]# 

Deployment配置文件

Deployment与Service等配置文件内容如下所示：

[root@host131 cadvisor]# cat deployment.yml 
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  namespace: default
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '9090'
  
spec:
  selector: 
    app: prometheus-deployment
  type: NodePort  
  ports:
    - port: 8080
      targetPort: 9090 
      nodePort: 33308
...
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-deployment
  template:
    metadata:
      labels:
        app: prometheus-deployment
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.15.1
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-configmap
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-configmap
          configMap:
            defaultMode: 420
            name: prometheus-configmap
  
        - name: prometheus-storage-volume
          emptyDir: {}
[root@host131 cadvisor]#

启动Prometheus服务

[root@host131 cadvisor]# ls
configmap.yml  deployment.yml  rbac.yml
[root@host131 cadvisor]# kubectl create -f .
configmap/prometheus-configmap created
service/prometheus-service created
deployment.apps/prometheus-deployment created
clusterrole.rbac.authorization.k8s.io/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
[root@host131 cadvisor]# 

结果确认

[root@host131 cadvisor]# kubectl get pods
NAME                                     READY   STATUS    RESTARTS   AGE
prometheus-deployment-774dcd78bc-v7gkv   1/1     Running   0          5s
[root@host131 cadvisor]# kubectl get deployments
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-deployment   1/1     1            1           10s
[root@host131 cadvisor]# kubectl get service
NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes           ClusterIP   10.254.0.1              443/TCP          88m
prometheus-service   NodePort    10.254.38.177           8080:33308/TCP   16s
[root@host131 cadvisor]# 

确认连接信息

确认cAdvisor版本

确认UP信息

从/targets链接也可以同样确认到UP的信息

服务发现的信息

常见问题

如果ConfigMap中的Prometheus.yml文件中的设定不正确，容易出现如下类似的错误信息（403 Forbidden）

配置文件

上述配置文件同样存放在Easypack中，URL地址链接为：

• https://github.com/liumiaocn/easypack/tree/master/monitor/prometheus/kubernetes/cadvisor