高可用Kuberbetes部署Prometheus + Grafana
概述
阅读官方文档部署部署Prometheus + Grafana
GitHub - prometheus-operator/kube-prometheus at release-0.10
环境
步骤
下周官方github仓库
git clone https://github.com/prometheus-operator/kube-prometheus.git
git checkout release-0.10进入工作目录
cd kube-prometheus/manifests
mkdir -p adapter alertmanager blackbox grafana kube-state-metrics node-exporter operator prometheus
修改镜像地址
主要是镜像难寻，全靠运气。
prometheusAdapter-deployment.yaml
修改为：thinkingdata/prometheus-adapter:v0.10.0 镜像可用
kubeStateMetrics-deployment.yaml
修改为：bitnami/kube-state-metrics:2.7.0 镜像可用
部署服务
根据官方文档部署服务
kubectl apply --server-side -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl apply -f manifests/验证
NodePort服务暴露
由ClusterIP修改为NodePort
网页访问验证：
只能在指定的节点访问服务。
如图是:
- node04访问grafana + Prometheus。
- master03访问Prometheus
默认登录账号密码都为 admin 登录就会要求你重设密码，重设密码仍为admin
Ingress服务暴露
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prom-ingress
namespace: monitoring
spec:
ingressClassName: nginx
rules:
- host: alert.k8s.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: alertmanager-main
port:
number: 9093
- host: grafana.k8s.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: grafana
port:
number: 3000
- host: prom.k8s.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: prometheus-k8s
port:
number: 9090
配置域名解析
配置Grafana
第一次登录账户密码都使用admin，详细教程参考：Grafana fundamentals | Grafana Labs
测试：
引入基本Dashboard
中文dashboard
K8S Dashboard CN 20211010 StarsL.cn | Grafana Labs
引入dashboard
导入刚才下载的json文件即可
查询其他Dashboard
Dashboards | Grafana Labs
成果：
配置邮件告警
查询grafana.ini配置在哪里
修改配置文件并重新部署kubectl apply -f grafana-config.yaml
Grafana配置SMTP接入邮箱 - 乱七八糟博客备份 - 博客园 (cnblogs.com)
grafana配置文件说明 - woaibaobei - 博客园 (cnblogs.com)
Grafana配置邮件告警_grafana配置邮件报警_lee_yanyi的博客-CSDN博客
删除grafana的原有pod，必须删除相当于重新加载配置
配置邮件发送SMTP其他邮件服务同理
测试联通性
安装插件
Error
ingress 部署失败
Error from server (InternalError): error when creating "ingress.yml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Posetworking/v1/ingresses?timeout=10s": dial tcp 10.96.222.96:443: connect: connection refused
删除错误pod
kubectl get pod -n ingress-nginx | grep Evicted | awk '{print $1}' | xargs kubectl delete pod -n ingress-nginx
重新部署pod
kubectl delete pod/ingress-nginx-controller-6ff65d977f-q2kw9 -n ingress-nginx
备注
使用kube-prometheus部署k8s监控(最新版)_净夜凡尘的博客-CSDN博客
Deploy Grafana on Kubernetes | Grafana documentation
kubernetes 部署Prometheus监控集群传统部署方案_ghostwritten的博客-CSDN博客
GitHub - starsliao/Prometheus: Grafana Dashboards for Prometheus Exporter