《kubernetes-1.8.0》09-addon-kube-prometheus

《kubernetes-1.8.0》09-addon-kube-prometheus

《kubernetes 1.8.0 测试环境安装部署》

时间：2017-11-23

一、环境准备

clone kube-prometheus 项目：

git clone https://github.com/coreos/prometheus-operator.git

指定kubeconfig目录位置（本例使用缺省，如果配置文件存放位置不是~/.kube/config需指定该变量）：

export KUBECONFIG= # defaults to "~/.kube/config"

二、安装kube-prometheus

运行kube-prometheus部署脚本：

$ cd prometheus-operator/contrib/kube-prometheus/
$ hack/cluster-monitoring/deploy
...
namespace "monitoring" created
clusterrolebinding "prometheus-operator" created
clusterrole "prometheus-operator" created
serviceaccount "prometheus-operator" created
service "prometheus-operator" created
deployment "prometheus-operator" created
Waiting for Operator to register custom resource definitions.................... 
daemonset "node-exporter" created
service "node-exporter" created
clusterrolebinding "kube-state-metrics" created
clusterrole "kube-state-metrics" created
deployment "kube-state-metrics" created
rolebinding "kube-state-metrics" created
role "kube-state-metrics-resizer" created
serviceaccount "kube-state-metrics" created
service "kube-state-metrics" created
secret "grafana-credentials" created
secret "grafana-credentials" configured
configmap "grafana-dashboards-0" created
deployment "grafana" created
service "grafana" created
servicemonitor "kube-state-metrics" created
servicemonitor "prometheus-operator" created
prometheus "k8s" created
servicemonitor "kubelet" created
servicemonitor "kube-controller-manager" created
serviceaccount "prometheus-k8s" created
servicemonitor "prometheus" created
servicemonitor "kube-apiserver" created
servicemonitor "alertmanager" created
servicemonitor "kube-scheduler" created
configmap "prometheus-k8s-rules" created
servicemonitor "node-exporter" created
service "prometheus-k8s" created
role "prometheus-k8s" created
role "prometheus-k8s" created
role "prometheus-k8s" created
clusterrole "prometheus-k8s" created
rolebinding "prometheus-k8s" created
rolebinding "prometheus-k8s" created
rolebinding "prometheus-k8s" created
clusterrolebinding "prometheus-k8s" created
secret "alertmanager-main" created
service "alertmanager-main" created
alertmanager "main" created

创建pod时间较长，镜像需要下比较久，其中 addon-resizer:1.0 在之前的镜像包中提供。

一个小插曲：

加载完毕后，30900端口访问失败，怀疑prometheus核心组件启动失败，但检查pod又都是Running。
查看日志后发现：

kube-controller-manage报错：
Event(v1.ObjectReference{Kind:”StatefulSet”, Namespace:”monitoring”, Name:”prometheus-k8s”, UID:”2fbbb615-d117-11e7-917e-005056bc52e5”, APIVersion:”apps”, ResourceVersion:”256502”, FieldPath:”“}): type: ‘Warning’ reason: ‘FailedCreate’ create Pod prometheus-k8s-0 in StatefulSet prometheus-k8s failed error: pods “prometheus-k8s-0” is forbidden: pod.Spec.SecurityContext.RunAsUser is forbidden

检查statefulset启动情况：

发现prometheus-k8s这个statefulset没有成功启动：
用describe查看：

排查半天，发觉可能是apiserver 的控制问题，检查apiserver配置文件：
去除：KUBE_ADMISSION_CONTROL中的SecurityContextDeny部分，分别重启apiserver。

再次检查statefulset和pod的情况：

[root@node-131 kube-prometheus]# kubectl get pods -n monitoring
NAME                                   READY     STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2       Running   0          16m
alertmanager-main-1                    2/2       Running   0          15m
alertmanager-main-2                    2/2       Running   0          15m
grafana-7d966ff57-twzx9                2/2       Running   0          17m
kube-state-metrics-b5f8d6c5-zvgh5      2/2       Running   0          17m
node-exporter-7zx6s                    1/1       Running   0          17m
node-exporter-b88t5                    1/1       Running   0          17m
node-exporter-dhx7x                    1/1       Running   0          17m
node-exporter-wnhvd                    1/1       Running   0          17m
prometheus-k8s-0                       2/2       Running   0          16m
prometheus-k8s-1                       2/2       Running   0          16m
prometheus-operator-66578f9cd9-wstrp   1/1       Running   0          17m

[root@node-131 kube-prometheus]# kubectl get statefulset -n monitoring
NAME                DESIRED   CURRENT   AGE
alertmanager-main   3         3         16m
prometheus-k8s      2         2         16m

SecurityContextDeny：具体的作用需查看官档。

三、测试访问：

访问prometheus（30900）：

访问alertmanager（30903）：

访问grafana（30902）：

至此kube-prometheus框架基本搭建完成，后续关于prometheus具体操作另写：

本系列其他内容：

• 01-环境准备

• 02-etcd群集搭建

• 03-kubectl管理工具

• 04-master搭建

• 05-node节点搭建

• 06-addon-calico

• 07-addon-kubedns

• 08-addon-dashboard

• 09-addon-kube-prometheus

• 10-addon-EFK

• 11-addon-Harbor

• 12-addon-ingress-nginx

• 13-addon-traefik

参考文档：

https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus