环境信息

Kubernetes：v1.20.6
StorageClass：csi-udisk-rssd
Helm：v3.5.2
nginx-ingress: 0.47.0

本安装前准备工作

1. 申请域名证书

可以使用 https://keymanager.org/ 来申请 Let’s Encrypt 提供的免费泛域名证书

2. 创建域名证书 secret

kubectl create ns monitor
kubectl create secret tls tls-prometheus-secret \
    --cert=domain.crt \
    --key=domain.key \
    -n monitor

3. 同步海外源镜像

在国内环境部署应用，经常因为获取国外源站容器镜像超时，导致部署失败，可以提前将容器镜像同步到本地镜像仓库中，以自有镜像仓库uhub.service.ucloud.cn/ucloud_pts 为例，login仓库，执行命令： docker login uhub.service.ucloud.cn/ucloud_pts
需要同步镜像列表如下：

prometheus:v2.26.0
configmap-reload:v0.5.0 
kube-state-metrics:v2.1.0 
node-exporter:v1.1.2

关于docker pull tag push 操作可以参考：

https://docs.docker.com/engine/reference/commandline/pull/
https://docs.docker.com/engine/reference/commandline/tag/
https://docs.docker.com/engine/reference/commandline/push/

4. 创建 imagePullSecrets

创建容器集群访问仓库地址 uhub.service.ucloud.cn/ucloud_pts，拉取镜像需要的 secret

kubectl create namespace monitor
kubectl create secret docker-registry registry-prometheus-secret \
        --namespace=monitor \
        --docker-server=uhub.service.ucloud.cn/ucloud_pts \
        --docker-username='xxxxxx' \
        --docker-password='xxxxxx'

4. 添加 Helm仓库

这里选用prometheus-community提供的chart仓库
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update

5. 选定Node节点添加标签

kubectl label nodes 10.1.10.218 prometheus=true --overwrite 
kubectl label nodes 10.1.121.197 prometheus=true --overwrite

6. 定义 prometheus 配置，完成 prometheus 部署

cat > prometheus-values.yaml << EOF
imagePullSecrets:
  - name: "harbor-secret"
alertmanager:
  enabled: false
pushgateway:
  enabled: false
kubeStateMetrics:
  enabled: true
kube-state-metrics:
  prometheusScrape: true
  image:
    repository: harbor.onwalk.net/pts/kube-state-metrics
    tag: v2.1.0
  imagePullSecrets:
    - name: "harbor-secret"
configmapReload:
  prometheus:
    enabled: true
    name: configmap-reload
    image:
      repository: harbor.onwalk.net/pts/configmap-reload
      tag: v0.5.0
nodeExporter:
  enabled: true
  image:
    repository: harbor.onwalk.net/pts/node-exporter
    tag: v1.1.2
server:
  enabled: true
  name: server
  image:
    repository: harbor.onwalk.net/pts/prometheus
    tag: latest
  baseURL: "http://prometheus.onwalk.net"
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hosts:
       - prometheus.onwalk.net
    tls:
      - secretName: prometheus.onwalk.net-tls
        hosts:
          - prometheus.onwalk.net
  persistentVolume:
    enabled: false
    accessModes:
      - ReadWriteOnce
    mountPath: /data
    size: 10Gi
    storageClass: csi-udisk-rssd
nodeSelector:
  prometheus: true
EOF
helm delete prometheus -n monitor
helm upgrade --install  prometheus prometheus-community/prometheus \
-f prometheus-values.yaml -n monitor

添加远端存储

kubectl edit cm -n monitor prometheus-server

  prometheus.yml: |
    global:
      external_labels:
        cluster: admin
    remote_write:
    - url: https://cortex-gateway.onwalk.net/api/prom/push

问题记录

prometheus 重建的时候，CRD资源如果不对用重建会丢失

 prometheus:
   prometheusSpec:
     enableRemoteWriteReceiver: true
     enableFeatures:
     - remote-write-receiver

helm 部署 prometheus