使用thanos、prometheus和minio打造高可用监控平台

在单kubernetes集群中，使用单实例或多副本的prometheus即可搭建好一个比较可用的监控平台。单该方案有很多的不足，有如下的缺点

a. 不能保存永久的监控数据（数据量太多会影响prometheus性能）
b. 多集群下grafana需要配置很多个数据源，比较麻烦
c. 节点异常会导致数据丢失， 不高可用

针对这些问题，目前的方案有：prometheus多副本+HA方案，数据量大是采用分层的prometheus联邦集群方案。终极方案：thanos

thanos部署方案有两种：sidecar和receive，github和官方文档都介绍得比较详细了。

Deployment with Sidecar:

image.png

Deployment with Receive:

image.png

k8s集群通过kind安装的，硬件为mac M1芯片

1. 通过kube-prometheus部署prometheus

https://github.com/prometheus-operator/kube-prometheus
下载好代码后，直接通过yaml的方式安装

kubectl apply --server-side -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl apply -f manifests/

2. 通过docker安装minio 对象存储

在kind环境下通过kubectl krew 安装minio时需要了问题，所以这儿通过docker来直接安装

docker run -d -p 9000:9000 -p 9001:9001 -v /Users/user/minio/data/:/data -v /Users/user/minio/config:/root/.minio minio/minio server --console-address ":9001" /data

通过过localhost:9001直接登录minio dashboard页面，9000端口给api用的，9001为dashboard
默认用户名密码为：minioadmin、minioadmin
登录后可以自己手动创建bucket

image.png

3. 通过kube-thanos安装thanos

https://github.com/search?q=kube-thanos
直接下载源码，通过yaml方式来安装。
由于thanos receive store等组件都是用statefulset来部署的，所以需要volume，这儿使用local pv
创建storageclass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
    name: local-storage
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

创建local pv

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /home
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kind-control-plane
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv1
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /home
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kind-control-plane

创建minio secrets

apiVersion: v1
kind: Secret
metadata:
  name: thanos-objectstorage
  namespace: thanos
type: Opaque
stringData:
  thanos.yaml: |-
    type: s3
    config:
      bucket: thanos
      endpoint: 192.168.0.102:9000 # 这个ip为上面安装的minio的ip+port，也就是我电脑的ip地址
      insecure: false
      access_key: minioadmin
      secret_key: minioadmin

kubectl create ns thanos
kubectl create -f manifests/

安装完后后，查看部署的内容如下：

kubectl get sc
NAME                      PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-storage (default)   kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  7h39m

kubectl get pv -n thanos
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                           STORAGECLASS    REASON   AGE
local-pv    10Gi       RWO            Delete           Bound    thanos/data-thanos-receive-ingestor-default-0   local-storage            4h22m
local-pv1   10Gi       RWO            Delete           Bound    thanos/data-thanos-store-0                      local-storage            4h22m
kubectl get pvc -n thanos
NAME                                     STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS    AGE
data-thanos-receive-ingestor-default-0   Bound    local-pv    10Gi       RWO            local-storage   4h23m
data-thanos-store-0                      Bound    local-pv6   10Gi       RWO            local-storage   4h23m

kubectl get pods -n thanos
NAME                                     READY   STATUS    RESTARTS   AGE
thanos-query-778c77b5-w6jjc              1/1     Running   0          4h21m
thanos-receive-ingestor-default-0        1/1     Running   0          4h21m
thanos-receive-router-557ddd87b9-4bb95   1/1     Running   0          4h21m
thanos-store-0                           1/1     Running   0          4h21m

指定port-forward打开thanos的页面
kubectl port-forward svc/thanos-query 9090:9090
配置prometheus的remotewrite为thanos receive的地址

# kube-prometheus/manifests/prometheus-prometheus.yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.32.1
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - apiVersion: v2
      name: alertmanager-main
      namespace: monitoring
      port: web
  enableFeatures: []
  externalLabels: {}
  image: quay.io/prometheus/prometheus:v2.32.1
  nodeSelector:
    kubernetes.io/os: linux
  podMetadata:
    labels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
      app.kubernetes.io/version: 2.32.1
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 1
  retention: 10d
  resources:
    requests:
      memory: 200Mi
  ruleNamespaceSelector: {}
  ruleSelector: {}
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: 2.32.1
  remoteWrite: #加入remoteWrite地址，注意替换为自己的地址
  - url: "http://thanos-receive-router-ip:19291/api/v1/receive"

打开thanos query的页面

image.png

数据中的cluster、datacenter字段为promtheus中的external_label字段，在多个集群接入thanos时，用来区分不同的集群，非常有用。filed中的tenant_id从remoteWrite的header字段中提取，支持多租户。