Kind & Kubernetes | 通过 Helm 部署定制化 Prometheus-Operator 上传 Dockerhub？

1. 环境准备

• 安装系统 Centos 8.2
• 初始化 Centos 8.2
• 安装 Podman，它是 Podman 最初由红帽工程师联合开源社区一同开发的无守护进程的下一代容器管理工具，即 Docker 替代者。
• 安装 Kubectl
  安装 Kind， 它是一个 Kubernetes 孵化项目，一套开箱即用的 Kubernetes 环境搭建方案。顾名思义，就是将 Kubernetes 所需要的所有组件，全部部署在一个 Docker 容器中。

curl -Lo ./kind https://github.com/kubernetes-sigs/kind/releases/download/v0.17.0/kind-linux-amd64
chmod +x ./kind
mv ./kind /usr/local/bin/kind

• 安装 helm 命令

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

或者

$ wget https://get.helm.sh/helm-v3.11.0-linux-amd64.tar.gz
$ tar -xzvf helm-v3.11.0-linux-amd64.tar.gz
$ cp linux-amd64/helm /usr/local/bin/
$ helm version
version.BuildInfo{Version:"v3.11.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}

2. Kind 部署 Kubernetes

• config.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP

创建 K8s 集群:

$ kind create cluster --config config.yaml
enabling experimental podman provider
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 
 ✓ Preparing nodes 
 ✓ Writing configuration 
 ✓ Starting control-plane ️
 ✓ Installing CNI 
 ✓ Installing StorageClass 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 

2.1 安装 Ingress-nginx 组件

kubectl create -f https://ghproxy.com/https://raw.githubusercontent.com/Ghostwritten/resource/main/ingress-nginx/ingress-nginx.yaml

2.2 安装 Metric Server 组件

系统资源的采集均使用Metrics-Server服务，可以通过Metrics-Server服务采集节点和Pod的内存、磁盘、CPU和网络的使用率等信息。Metric Server组件是实现服务自动扩容不可或缺的组件。

kubectl apply -f https://ghproxy.com/https://raw.githubusercontent.com/Ghostwritten/resource/main/metrics/metrics.yaml

等待 Metric 工作负载就绪

kubectl wait deployment -n kube-system metrics-server --for condition=Available=True --timeout=90s

Metric Server 就绪后，我们通过 kubectl autoscale 命令来为 Deployment 创建自动扩容策略。但这篇不是重点。

3. helm 快速安装 Prometheus-Operator

默认安装方式：

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm upgrade prometheus prometheus-community/kube-prometheus-stack \--namespace prometheus  --create-namespace --install \--set prometheusOperator.admissionWebhooks.patch.image.registry=docker.io --set prometheusOperator.admissionWebhooks.patch.image.repository=dyrnq/kube-webhook-certgen \--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false 

Release "prometheus" does not exist. Installing it now.
NAME: prometheus
LAST DEPLOYED: Tue Mar 28 16:22:32 2023
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace prometheus get pods -l "release=prometheus"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

虽然部署成功了，但是查看 pod 报以下错误：

$k get pod -n prometheus
NAME                                                     READY   STATUS         RESTARTS      AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running        1 (33s ago)   36s
prometheus-grafana-6f77bc5bc9-9qkf6                      3/3     Running        0             41s
prometheus-kube-prometheus-operator-6d7bf45ccc-mlnzm     1/1     Running        0             41s
prometheus-kube-state-metrics-678b896dcf-ntz5r           0/1     ErrImagePull   0             41s
prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running        0             33s
prometheus-prometheus-node-exporter-4dxf5                1/1     Running        0             41s

$ k describe pod -n prometheus prometheus-kube-state-metrics-678b896dcf-2qvwt
.......
  Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  36s   default-scheduler  Successfully assigned prometheus/prometheus-kube-state-metrics-678b896dcf-2qvwt to kind-control-plane
  Normal   Pulling    36s   kubelet            Pulling image "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0"
  Warning  Failed     5s    kubelet            Failed to pull image "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0": failed to resolve reference "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0": failed to do request: Head "https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/kube-state-metrics/kube-state-metrics/manifests/v2.8.0": dial tcp 142.251.170.82:443: i/o timeout
  Warning  Failed     5s    kubelet            Error: ErrImagePull
  Normal   BackOff    5s    kubelet            Back-off pulling image "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0"
  Warning  Failed     5s    kubelet            Error: ImagePullBackOff
........

没有真正的全球互联网，我们无法拉取 registry.k8s.io。需要对重新定制自己的 prometheus helm charts。

3.1 获取镜像方法

但由于隐形墙的原因，我们无法下载拉取以下：

• registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0

所以，你可以获取registry.k8s.io的方法：

• 购买国外 VPS 拉取镜像推送至 DockerHub
• 通过配置代理拉取镜像，需施展要魔法技能。
• 去 DockerHub、quay.io 寻找他人推送的对应版本的镜像。
• 利用 Github Action 实现镜像自动迁移。fork来自@togettoyou YYDS

这里我利用 Github Action 实现镜像自动迁移。

3.2 定制内容

• 下载最新版本(20230328) kube-prometheus-stack-45.8.0.tgz

wget https://github.com/prometheus-community/helm-charts/releases/download/kube-prometheus-stack-45.8.0/kube-prometheus-stack-45.8.0.tgz
tar -zxvf kube-prometheus-stack-45.8.0.tgz

修改镜像版本

• registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.8.0改为docker.io/ghostwritten/kube-state-metrics:v2.8.0

首先：vim charts/kube-state-metrics/values.yaml

  3 image:
  4   registry: registry.k8s.io
  5   repository: kube-state-metrics/kube-state-metrics
  6   # If unset use v + .Charts.appVersion
  7   tag: ""

修改为：

  3 image:
  4   registry: docker.io
  5   repository: ghostwritten/registry.k8s.io.kube-state-metrics.kube-state-metrics
  6   # If unset use v + .Charts.appVersion
  7   tag: "v2.8.0"

第二步：为了和官方 prometheus helm charts 做一个区分需要修改一下 helm charts的名字。打开vim Chart.yaml

 50 name: kube-prometheus-stack

修改为：

 50 name: ghostwritten-kube-prometheus-stack

3.3 打包

$ cd ../
$ helm package ./kube-prometheus-stack
Successfully packaged chart and saved it to: /root/ghostwritten-kube-prometheus-stack-45.2.0.tgz

3.4 推送 Helm Package 至 DockerHub

如何上传 Github 或者 Harbor 请参考这篇文章：kind & kubernetes 集群内如何通过 helm 部署定制化 Prometheus-Operator？

创建 Access token

$ export REG_PAT=<token>

$  echo $REG_PAT | helm registry login registry-1.docker.io -u ghostwritten --password-stdin
Login Succeeded

$ helm push ghostwritten-kube-prometheus-stack-45.8.0.tgz oci://registry-1.docker.io/ghostwritten
Pushed: registry-1.docker.io/ghostwritten/ghostwritten-kube-prometheus-stack:45.8.0
Digest: sha256:20bb9e1e22d5bb422bd285a829061b58458adde6ebc26f5ee81cca8f31ef0729

登陆 DockerHub 验证是否上传成功。

4. helm 安装定制化的 ghostwritten-kube-prometheus-stack

我这里选择从刚刚上传的公共 dockerhub上的ghostwritten-kube-prometheus-stack包进行安装。

helm upgrade prometheus oci://registry-1.docker.io/ghostwritten/ghostwritten-kube-prometheus-stack  --version 45.8.0 \--namespace prometheus  --create-namespace --install \--set prometheusOperator.admissionWebhooks.patch.image.registry=docker.io --set prometheusOperator.admissionWebhooks.patch.image.repository=dyrnq/kube-webhook-certgen \--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false 

输出：

Release "prometheus" does not exist. Installing it now.
Pulled: registry-1.docker.io/ghostwritten/ghostwritten-kube-prometheus-stack:45.8.0
Digest: sha256:20bb9e1e22d5bb422bd285a829061b58458adde6ebc26f5ee81cca8f31ef0729
NAME: prometheus
LAST DEPLOYED: Tue Mar 28 17:28:51 2023
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
NOTES:
ghostwritten-kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace prometheus get pods -l "release=prometheus"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

接下来，等待 Prometheus 所有组件处于 Ready 状态。

$ kubectl wait --for=condition=Ready pods --all -n prometheus --timeout=300s
pod/alertmanager-prometheus-ghostwritten-ku-alertmanager-0 condition met
pod/prometheus-ghostwritten-ku-operator-67d79fc6f7-cx5kk condition met
pod/prometheus-grafana-79486fb946-mhng2 condition met
pod/prometheus-kube-state-metrics-85c6bbbcb-c7kms condition met
pod/prometheus-prometheus-ghostwritten-ku-prometheus-0 condition met
pod/prometheus-prometheus-node-exporter-5bdrl condition met

$  k get pods -n prometheus
NAME                                                     READY   STATUS    RESTARTS      AGE
alertmanager-prometheus-ghostwritten-ku-alertmanager-0   2/2     Running   1 (72s ago)   75s
prometheus-ghostwritten-ku-operator-67d79fc6f7-cx5kk     1/1     Running   0             79s
prometheus-grafana-79486fb946-mhng2                      3/3     Running   0             79s
prometheus-kube-state-metrics-85c6bbbcb-c7kms            1/1     Running   0             79s
prometheus-prometheus-ghostwritten-ku-prometheus-0       2/2     Running   0             72s
prometheus-prometheus-node-exporter-5bdrl                1/1     Running   0             79s

当所有 Pod 准备就绪后，代表 helm 部署的最新版本 kube-prometheus-stack 已经安装完成了。