Prometheus+Grafana（外）监控Kubernetes(K8s)集群(基于containerd)
一、实验环境
1、k8s环境
版本
v1.26.5
二进制安装Kubernetes(K8s)集群(基于containerd)—从零安装教程（带证书）
| 主机名 | IP | 系统版本 | 安装服务 |
|---|---|---|---|
| master01 | 10.10.10.21 | rhel7.5 | nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy |
| master02 | 10.10.10.22 | rhel7.5 | nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy |
| master03 | 10.10.10.23 | rhel7.5 | nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy |
| node01 | 10.10.10.24 | rhel7.5 | nginx、kubelet、proxy |
| node02 | 10.10.10.25 | rhel7.5 | nginx、kubelet、proxy |
2、Prometheus+Grafana环境
Prometheus+Grafana监控系统
| 主机名 | IP | 系统版本 |
|---|---|---|
| jenkins | 10.10.10.10 | rhel7.5 |
3、Prometheus部署方式
- kubernetes内部Prometheus监控k8s集群
- Prometheus监控内部K8S就是把Prometheus部署在K8S集群内,比如部署在K8S集群的monitoring的namespace下,因为K8S在所有的namespace下自动创建了serviceAccount和对应的Secret里自带访问K8S API的token和ca,所以就不需要手动创建serviceAccount和Secret了
- kubernetes外部Prometheus监控k8s
- kubernetes外部Prometheus监控外部K8S就是把Prometheus部署在虚拟机上，需要自己在Prometheus.yaml手动指定API的地址，ca和Token
4、版本对应
https://github.com/kubernetes/kube-state-metrics
二、配置kube-state-metrics
https://github.com/kubernetes/kube-state-metrics/tree/v2.9.2/examples/standard
1、文件下载
[root@master01 kube-state-metrics]# ls
cluster-role-binding.yaml cluster-role.yaml deployment.yaml service-account.yaml service.yaml
[root@master01 kube-state-metrics]# cat cluster-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/version: 2.9.2
name: kube-state-metrics
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-state-metrics
subjects:
- kind: ServiceAccount
name: kube-state-metrics
namespace: kube-system
[root@master01 kube-state-metrics]# cat cluster-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/version: 2.9.2
name: kube-state-metrics
rules:
- apiGroups:
- ""
resources:
- configmaps
- secrets
- nodes
- pods
- services
- serviceaccounts
- resourcequotas
- replicationcontrollers
- limitranges
- persistentvolumeclaims
- persistentvolumes
- namespaces
- endpoints
verbs:
- list
- watch
- apiGroups:
- apps
resources:
- statefulsets
- daemonsets
- deployments
- replicasets
verbs:
- list
- watch
- apiGroups:
- batch
resources:
- cronjobs
- jobs
verbs:
- list
- watch
- apiGroups:
- autoscaling
resources:
- horizontalpodautoscalers
verbs:
- list
- watch
- apiGroups:
- authentication.k8s.io
resources:
- tokenreviews
verbs:
- create
- apiGroups:
- authorization.k8s.io
resources:
- subjectaccessreviews
verbs:
- create
- apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- list
- watch
- apiGroups:
- certificates.k8s.io
resources:
- certificatesigningrequests
verbs:
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
- volumeattachments
verbs:
- list
- watch
- apiGroups:
- admissionregistration.k8s.io
resources:
- mutatingwebhookconfigurations
- validatingwebhookconfigurations
verbs:
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- networkpolicies
- ingressclasses
- ingresses
verbs:
- list
- watch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- list
- watch
- apiGroups:
- rbac.authorization.k8s.io
resources:
- clusterrolebindings
- clusterroles
- rolebindings
- roles
verbs:
- list
- watch
[root@master01 kube-state-metrics]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/version: 2.9.2
name: kube-state-metrics
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: kube-state-metrics
template:
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/version: 2.9.2
spec:
automountServiceAccountToken: true
containers:
- image: bitnami/kube-state-metrics:2.9.2
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
timeoutSeconds: 5
name: kube-state-metrics
ports:
- containerPort: 8080
name: http-metrics
- containerPort: 8081
name: telemetry
readinessProbe:
httpGet:
path: /
port: 8081
initialDelaySeconds: 5
timeoutSeconds: 5
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: kube-state-metrics
[root@master01 kube-state-metrics]# cat service-account.yaml
apiVersion: v1
automountServiceAccountToken: false
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/version: 2.9.2
name: kube-state-metrics
namespace: kube-system
[root@master01 kube-state-metrics]# cat service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: exporter
app.kubernetes.io/name: kube-state-metrics
app.kubernetes.io/version: 2.9.2
name: kube-state-metrics
namespace: kube-system
spec:
type: NodePort
ports:
- name: http-metrics
port: 8080
targetPort: 8080
nodePort: 32080
protocol: TCP
- name: telemetry
port: 8081
targetPort: 8081
nodePort: 32081
protocol: TCP
selector:
app.kubernetes.io/name: kube-state-metrics
2、安装kube-state-metrics
使用NodePort暴漏端口
[root@master01 kube-state-metrics]# kubectl apply -f ./
[root@master01 kube-state-metrics]# kubectl get po -n kube-system -o wide | grep kube-state-metrics
kube-state-metrics-57ddc8c4ff-krsh2 1/1 Running 0 9m5s 10.0.3.1 master02
[root@master01 kube-state-metrics]# kubectl get svc -n kube-system | grep kube-state-metrics
kube-state-metrics NodePort 10.97.38.90 8080:32080/TCP,8081:32081/TCP 9m17s
3、测试结果
发现部署在master02，也就是10.10.10.22
[root@master01 kube-state-metrics]# curl http://10.97.38.90:8080/healthz -w '\n'
OK
三、配置Prometheus
1、修改prometheus.yml
[root@jenkins ~]# cat Prometheus/prometheus.yml
- job_name: "kube-state-metrics"
static_configs:
- targets: ["10.10.10.22:32080"]
- job_name: "kube-state-telemetry"
static_configs:
- targets: ["10.10.10.22:32081"]
2、重启Prometheus
[root@jenkins ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a0497377cd82 grafana/grafana-enterprise "/run.sh" 13 days ago Up 3 minutes 0.0.0.0:3000->3000/tcp grafana
3e0e4270bd92 prom/prometheus "/bin/prometheus --c…" 13 days ago Up 3 minutes 0.0.0.0:9090->9090/tcp prometheus
[root@jenkins ~]# docker restart 3e0e4270bd92
3、登录查看结果
四、配置Grafana
推荐模板：13332、13824、14518
1、导入模板
2、查看结果
