K8S-1.18.20高可用集群之部署集群监控系统kube-prometheus插件

K8S-1.18.20高可用集群之部署集群插件-KUBE-PROMETHEUS插件

一、简介

kube-prometheus 是一整套监控解决方案，它使用 Prometheus 采集集群指标，Grafana 做展示，包含如下组件：

The Prometheus Operator
Highly available Prometheus
Highly available Alertmanager
Prometheus node-exporter
Prometheus Adapter for Kubernetes Metrics APIs （k8s-prometheus-adapter）
kube-state-metrics
Grafana
其中 k8s-prometheus-adapter 使用 Prometheus 实现了 metrics.k8s.io 和 custom.metrics.k8s.io API，所以不需要再部署 metrics-server。
如果没有特殊指明，本文档的所有操作均在 k8s-Mater 节点上执行；

二、下载kube-prometheus部署yaml

2.1、选择kube-prometheus版本

最新版本参考kube-prometheus git仓库：https://github.com/prometheus-operator/kube-prometheus

2.2、下载K8S版本对应的kube-prometheus

这里下载kube-prometheus 0.5.0
下载地址：https://codeload.github.com/prometheus-operator/kube-prometheus/zip/refs/tags/v0.5.0

三、修改yaml文件使用阿里云镜像

3.1、查看部署Yaml所需要的镜像

[root@localhost opt]# grep -r image: ./manifests
./manifests/alertmanager-alertmanager.yaml:  image: quay.io/prometheus/alertmanager:v0.20.0
./manifests/grafana-deployment.yaml:        image: grafana/grafana:6.6.0
./manifests/kube-state-metrics-deployment.yaml:        image: quay.io/coreos/kube-state-metrics:v1.9.5
./manifests/kube-state-metrics-deployment.yaml:        image: quay.io/coreos/kube-rbac-proxy:v0.4.1
./manifests/kube-state-metrics-deployment.yaml:        image: quay.io/coreos/kube-rbac-proxy:v0.4.1
./manifests/node-exporter-daemonset.yaml:        image: quay.io/prometheus/node-exporter:v0.18.1
./manifests/node-exporter-daemonset.yaml:        image: quay.io/coreos/kube-rbac-proxy:v0.4.1
./manifests/prometheus-adapter-deployment.yaml:        image: quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0
./manifests/prometheus-prometheus.yaml:  image: quay.io/prometheus/prometheus:v2.15.2
./manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml:                  image:
./manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml:            image:
./manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml:                  image:
./manifests/setup/prometheus-operator-0alertmanagerCustomResourceDefinition.yaml:                      image:
./manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml:                  image:
./manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml:            image:
./manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml:                  image:
./manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml:                image:
./manifests/setup/prometheus-operator-0prometheusCustomResourceDefinition.yaml:                      image:
./manifests/setup/prometheus-operator-0thanosrulerCustomResourceDefinition.yaml:                  image:
./manifests/setup/prometheus-operator-0thanosrulerCustomResourceDefinition.yaml:            image:
./manifests/setup/prometheus-operator-0thanosrulerCustomResourceDefinition.yaml:                  image:
./manifests/setup/prometheus-operator-0thanosrulerCustomResourceDefinition.yaml:                      image:
./manifests/setup/prometheus-operator-deployment.yaml:        image: quay.io/coreos/prometheus-operator:v0.38.1
./manifests/setup/prometheus-operator-deployment.yaml:        image: quay.io/coreos/kube-rbac-proxy:v0.4.1

[root@localhost opt]# grep -r "image: grafana" ./manifests
./manifests/grafana-deployment.yaml:        image: grafana/grafana:6.6.0

3.2、替换yaml中镜像地址为阿里云镜像

3.2.1、替换quay.io镜像

[root@localhost opt]# sed -i "s/quay.io\/prometheus/registry.cn-hangzhou.aliyuncs.com\/wc181/g" `grep "quay.io" -rl ./`
[root@localhost opt]# sed -i "s/quay.io\/coreos/registry.cn-hangzhou.aliyuncs.com\/wc181/g" `grep "quay.io" -rl ./`
[root@localhost opt]# sed -i "s/grafana\/prometheus/registry.cn-hangzhou.aliyuncs.com\/wc181/g" `grep "quay.io" -rl ./`

3.2.3、kube-prometheus需要用到的镜像

docker pull registry.cn-hangzhou.aliyuncs.com/wc181/alertmanager:v0.20.0
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/grafana:6.6.0
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/kube-state-metrics:v1.9.5
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/kube-rbac-proxy:v0.4.1
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/node-exporter:v0.18.1
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/k8s-prometheus-adapter-amd64:v0.5.0
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/prometheus:v2.15.2
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/prometheus-operator:v0.38.1
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/configmap-reload:v0.3.0
docker pull registry.cn-hangzhou.aliyuncs.com/wc181/prometheus-config-reloader:v0.38.1

以上项目部署在monitoring 名称空间下
验证部署情况

[root@dfycb-master1 setup]# kubectl get pod -o wide -n monitoring
NAME                                   READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
alertmanager-main-0                    2/2     Running   0          7h57m   10.70.7.71    dfycb-node5                
alertmanager-main-1                    2/2     Running   0          7h57m   10.70.6.55    dfycb-node4                
alertmanager-main-2                    2/2     Running   0          7h57m   10.70.5.66    dfycb-node3                
grafana-7469c5dc7d-bffpr               1/1     Running   0          8h      10.70.7.68    dfycb-node5                
kube-state-metrics-74b48b4bb7-lsnbl    3/3     Running   0          8h      10.70.3.60    dfycb-node1                
node-exporter-4sdbn                    2/2     Running   0          8h      10.60.6.147   dfycb-node8                
node-exporter-628bp                    2/2     Running   0          8h      10.60.6.142   dfycb-node3                
node-exporter-8jc9p                    2/2     Running   0          8h      10.60.6.139   dfycb-master3              
node-exporter-f9zxr                    2/2     Running   0          8h      10.60.6.140   dfycb-node1                
node-exporter-gghpn                    2/2     Running   0          8h      10.60.6.143   dfycb-node4                
node-exporter-hfg9j                    2/2     Running   0          8h      10.60.6.138   dfycb-master2              
node-exporter-nggm2                    2/2     Running   0          8h      10.60.6.137   dfycb-master1              
node-exporter-qfnmr                    2/2     Running   0          8h      10.60.6.145   dfycb-node6                
node-exporter-vjsx4                    2/2     Running   0          8h      10.60.6.146   dfycb-node7                
node-exporter-vm8t4                    2/2     Running   0          8h      10.60.6.141   dfycb-node2                
node-exporter-zphp4                    2/2     Running   0          8h      10.60.6.144   dfycb-node5                
prometheus-adapter-6fb854c766-22fp5    1/1     Running   0          8h      10.70.6.53    dfycb-node4                
prometheus-k8s-0                       3/3     Running   1          7h57m   10.70.5.67    dfycb-node3                
prometheus-k8s-1                       3/3     Running   1          7h57m   10.70.9.68    dfycb-node7                
prometheus-operator-65549d9758-jhqz9   2/2     Running   0          7h57m   10.70.7.70    dfycb-node5                

四、部署kube-prometheus

4.1、部署kube-prometheus

[root@k8s01 manifests]# kubectl apply -f setup/*
[root@k8s01 manifests]# kubectl apply -f ./*

4.2、通过ingress暴露prometheus、grafana

prometheus

grafana

4.3、网页验证是否可以正常打开

prometheus UI

grafana

五、grafana展示

5.1、添加数据源

默认已添加

5.2、导入展示面板

通过ID导入：8919
也可以通过连接下载导入json
https://grafana.com/grafana/dashboards/8919-1-node-exporter-for-prometheus-dashboard-cn-0413-consulmanager/?tab=revisions

5.3、导入8919

因为是内网环境无法通过互联网导入，这里选择先下载json再导入
下载地址：https://grafana.com/api/dashboards/8919/revisions/14/download
文件：1-node-exporter-for-prometheus-dashboard-cn-0413-consulmanager_rev14
下载版本：14
最后展示效果：

文章参考：https://www.freesion.com/article/8614604815/