工作随笔 - k8s离线安装存储/中间件/CICD等组件

前提是k8s已经部署好，具体安装步骤参考https://www.jianshu.com/p/98b352f16223。存储不再使用NFC，替代方案longhorn。

安装Helm，方便快速安装组件

# helm安装
tar -xzvf helm-v3.5.4-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/ && chmod +x /usr/local/bin/helm
helm version

安装longhorn存储来作为PV使用

# 安装依赖
yum -y install iscsi-initiator-utils
systemctl enable iscsid
systemctl start iscsid
yum -y install nfs-utils

# 下载helm模板
helm repo add longhorn https://charts.longhorn.io
helm search repo longhorn/longhorn  -l
helm fetch longhorn/longhorn --version v1.2.4

# 解压
tar -xzvf longhorn-1.2.4.tgz

# 准备离线镜像
harbor.test.lo:5000/longhornio/longhorn-engine:v1.2.4
harbor.test.lo:5000/longhornio/longhorn-manager:v1.2.4
harbor.test.lo:5000/longhornio/longhorn-ui:v1.2.4
harbor.test.lo:5000/longhornio/longhorn-instance-manager:v1_20220303
harbor.test.lo:5000/longhornio/longhorn-share-manager:v1_20211020
harbor.test.lo:5000/longhornio/backing-image-manager:v2_20210820
harbor.test.lo:5000/longhornio/csi-attacher:v3.2.1
harbor.test.lo:5000/longhornio/csi-provisioner:v2.1.2
harbor.test.lo:5000/longhornio/csi-node-driver-registrar:v2.3.0
harbor.test.lo:5000/longhornio/csi-resizer:v1.2.0
harbor.test.lo:5000/longhornio/csi-snapshotter:v3.0.3

# --set defaultSettings.createDefaultDiskLabeledNodes=true 设置只有添加标签的才会设置为存储节点，默认把worker节点作为longhorn可调度节点
kubectl label nodes {worker-node-4,worker-node-5,worker-node-64} node.longhorn.io/create-default-disk=true

# Harbor允许匿名拉取镜像，不需要配置harbor仓库的账号密码。注意修改本地的path:/home/test/longhorn
helm install longhorn ./longhorn --namespace longhorn --create-namespace --set defaultSettings.createDefaultDiskLabeledNodes=true --set defaultSettings.defaultDataPath="/home/test/longhorn" --set defaultSettings.priority-class=high-priority --set defaultSettings.taintToleration="app=longhorn:NoSchedule" --set defaultSettings.defaultReplicaCount=2 --set service.ui.type=NodePort --set service.ui.nodePort=30001 --set privateRegistry.registryUrl=harbor.test.lo:5000

# 如果Harbor需要账号密码访问
# 创建默认拉取harbor镜像的secret
kubectl create secret docker-registry harbor-secret \
  --docker-server=harbor.test.lo:5000 \
  --docker-username=admin \
  --docker-password=pwd4test \
  -n longhorn

# 安装,设置harbor地址，设置准备存储数据的目录（注意不要 挂载到根目录/longhorn ），
helm install longhorn ./longhorn --namespace longhorn --create-namespace --set defaultSettings.createDefaultDiskLabeledNodes=true --set defaultSettings.defaultDataPath="/home/test/longhorn" --set defaultSettings.priority-class=high-priority --set defaultSettings.taintToleration="app=longhorn:NoSchedule" --set defaultSettings.defaultReplicaCount=2 --set service.ui.type=NodePort --set service.ui.nodePort=30001 --set privateRegistry.registryUrl=harbor.test.lo:5000 --set privateRegistry.registryUser=admin --set privateRegistry.registryPasswd=pwd4test  --set privateRegistry.registrySecret=harbor-secret

# 查看资源是否启动成功
kubectl get all -n longhorn

# 控制台(由于nodeport暴露端口30001)
http://192.168.0.1:30001/

Jenkins部署，以及流水线配置

1. 提前准备离线镜像

harbor.test.lo:5000/middleware/jenkins:2.365
harbor.test.lo:5000/base/jnlp-slave:4.13.2-1-jdk11
harbor.test.lo:5000/base/maven:3.5.3
harbor.test.lo:5000/base/node:lts
harbor.test.lo:5000/base/alpine:tools
harbor.test.lo:5000/base/helm:v3.7.0-p
harbor.test.lo:5000/base/docker:19.03.9-git

2. 为减少构建时间，需将部分数据持久化，创建pvc：

# 创建namespace
kubectl create ns jenkins
# 创建PVC
cat<

3. docker方式启动Jenkins Master节点（方式一）

# docker方式启动docker master节点，50000端口为slave连接master端口
mkdir -p /home/finance/Data/Jenkins && chmod -R 777 /home/finance/Data/Jenkins
docker run -d --name=jenkins --restart=always -e JENKINS_HTTP_PORT_NUMBER=8080 -p 18081:8080 -p 50000:50000 -v /home/finance/Data/Jenkins:/var/jenkins_home harbor.test.lo:5000/middleware/jenkins:2.365
# 验证Jenkins，创建账号，修改密码
http://192.168.0.1:18081
# 安装插件 (插件提前在有公网环境的容器中拷贝出来)
tar -xzvf /home/finance/packages/plugins.tgz -C /tmp/
docker cp /tmp/plugins jenkins:/var/jenkins_home/
docker restart jenkins

4. k8s方式启动Jenkins Master节点（方式二）

# k8s方式启动docker master节点
# 给node打标签，固定jenkins的pod到此几点
kubectl label nodes worker-node-1 build=true
# 启动jenkins
cat<

5. 配置Jenkins使用动态的k8s中Jenkins Slave节点

登陆Jenkins -> 系统管理 -> Manage Credentials -> ## Stores scoped to Jenkins -> 全局凭据 (unrestricted) -> 添加凭证

• 添加凭证，Harbor的账号密码、 GitLab 的私钥、 Kubernetes 的证书均使用 Jenkins 的 Credentials 管理。
  示例凭据信息：
• 类型: SSH Username with password；默认代码仓库ID: GITLAB;
• 类型: SSH Username with password；默认HARBOR仓库ID: HARBOR_ACCOUNT;
• 类型: secret-file;默认K8S证书凭证ID: K8S-STANDARD;文件选择：上传~/.kube/config

登陆Jenkins -> 全局安全配置 -> TCP port for inbound agents -> 指定端口（50000）备注：需要和jenkins master的service中暴露的端口对应，用于slave节点连接master节点

Jenkins配置Slave节点

Jenkins的slave节点，可以通过标签控制调度到某个k8s worker节点

kubectl label node  build=true

登陆Jenkins -> 系统管理 -> 节点管理 -> Configure Clouds -> 配置集群 -> kubernetes Cloud Details

• name: standard (需要和Jenkinsfile中agent下cloud 'standard'对应)
• Kubernetes 地址: http://k8svip.test.lo:6443（apiserver地址）
• Kubernetes 命名空间: Jenkins （slave节点调度的命名空间）
• 凭据：config（选择刚刚创建的k8s证书凭证）
• Jenkins 地址：http://jenkins-master:8080（master节点的service地址，如果容器版本master节点，请适当修改）
  点击连接测试

6. 在代码仓库中配置Jenkinsfile和Dockerfile

Jenkinsfile具体内容示例如下

pipeline {
  # 保留多少个流水线的构建记录配置
  options {
   buildDiscarder(logRotator(numToKeepStr: '2'))
  }
  # 定义使用 Kubernetes 作为 agent
  agent {
    kubernetes {
      # 选择的云为之前配置的名字
      cloud 'standard'
      # 将 workspace 改成 PVC 的模式，也可以使用改成 hostPath：workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)
      workspaceVolume persistentVolumeClaimWorkspaceVolume(claimName: 'jenkins-workspace', readOnly: false)
      yaml '''
apiVersion: v1
kind: Pod
spec:
  containers:
    # jnlp容器 ，和Jenkins主节点通信
    - args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']
      image: 'harbor.test.lo:5000/base/jnlp-slave:4.13.2-1-jdk11'
      name: jnlp
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
        - mountPath: "/etc/hosts"
          name: "volume-hosts"
          readOnly: false
    # build 容器，包含执行构建的命令， 比如Java的需要 mvn构建，就可以用一个maven的镜像，NodeJS应用就使用node镜像   
    - command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
      # 使用Maven镜像，包含mvn工具，NodeJS 可以用 node 的镜像
      image: "harbor.test.lo:5000/base/maven:3.5.3"
      imagePullPolicy: "IfNotPresent"
      name: "build"
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
       # Pod 单独创建了一个缓存的 volume ，将其挂载到了maven插件的缓存目录，默认是/root/.m2
        - mountPath: "/root/.m2/"
          name: "volume-maven-repo"
          readOnly: false
        - mountPath: "/etc/hosts"
          name: "volume-hosts"
          readOnly: false
        - mountPath: "/root/settings"
          name: settings
    # 变更容器，因为最终是发版至 Kubernetes 的，所以需要有一个helm命令来实现变更
    - command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
      image: "harbor.test.lo:5000/base/helm:v3.7.0-p"
      imagePullPolicy: "IfNotPresent"
      name: "helm"
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
        - mountPath: "/var/run/docker.sock"
          name: "volume-docker"
          readOnly: false
        - mountPath: "/mnt/.kube/"
          name: "volume-kubeconfig"
          readOnly: false
        - mountPath: "/etc/hosts"
          name: "volume-hosts"
          readOnly: false
    # 用于bulid镜像的容器，需要包含docker命令
    - command:
        - "cat"
      env:
        - name: "LANGUAGE"
          value: "en_US:en"
        - name: "LC_ALL"
          value: "en_US.UTF-8"
        - name: "LANG"
          value: "en_US.UTF-8"
      image: "harbor.test.lo:5000/base/docker:19.03.9-git"
      imagePullPolicy: "IfNotPresent"
      name: "docker"
      tty: true
      volumeMounts:
        - mountPath: "/etc/localtime"
          name: "volume-2"
          readOnly: false
        # 由于容器没有启动 docker 服务，所以将宿主机的 docker 经常挂载至容器即可
        - mountPath: "/var/run/docker.sock"
          name: "volume-docker"
          readOnly: false
        - mountPath: "/etc/hosts"
          name: "volume-hosts"
          readOnly: false
  restartPolicy: "Never"
  # 固定节点部署slave，和之前创建的标签需一致
  nodeSelector:
    build: "true"
  securityContext: {}
  volumes:
    - hostPath:
        path: "/var/run/docker.sock"
      name: "volume-docker"
    - hostPath:
        path: "/usr/share/zoneinfo/Asia/Shanghai"
      name: "volume-2"
    - hostPath:
        path: "/etc/hosts"
      name: "volume-hosts"
    - name: "volume-maven-repo"
      persistentVolumeClaim:
        claimName: "maven-m2"
    - name: "volume-kubeconfig"
      secret:
        # 由master节点下面的~/.kube/config创建的
        secretName: "multi-kube-config"
    - name: "settings"
      secret:
        # 由maven的/root/settings/settings.xml创建，内容包含认证maven仓库等信息
        secretName: "zkj-settings"
'''
  }
}
  stages {
    # 拉取指定分支代码
    stage('Pulling Code') {
      parallel {
        stage('Pulling Code by Jenkins') {
          when {
            expression {
              env.gitlabBranch == null
            }
             
        }
          steps {
            git(branch: "${BRANCH}", credentialsId: 'GITLAB', url: "${GIT_URL}")
            script {
              COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
              TAG = BUILD_ID+'_'+COMMIT_ID
              println "Env is ${ENV}, App_name is ${APP_NAME}, Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}, Giturl is ${GIT_URL}"
            }
          }
        }
        stage('Pulling Code by trigger') {
          when {
            expression {
              env.gitlabBranch != null
           }
         }
         steps {
           git(branch: "${BRANCH}", credentialsId: 'GITLAB', url: "${GIT_URL}")
           script {
              COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
              TAG = BUILD_ID+'_'+COMMIT_ID
              println "Env is ${ENV}, App_name is ${APP_NAME}, Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}, Giturl is ${GIT_URL}"
            }
          }
        }
      }
    }
   
  # 构建应用
  stage('Building') {
    steps {
      container(name: 'build') {
        sh """
          mvn clean package -U -Dmaven.test.skip=true -gs /root/settings/settings.xml   #构建命令
          ls target/*
          """
          }
        }
    }
     
  #构建镜像并推送到代码仓库
  stage('Docker build for creating image') {
    environment {
      HARBOR_USER = credentials('HARBOR_ACCOUNT')
    }
    steps {
      container(name: 'docker') {
        sh """
          echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG} ${ENV} ${APP_NAME}
          docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${APP_NAME}:${TAG} .   #构建镜像
          docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}
          docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${APP_NAME}:${TAG}  #push镜像
          """
        }
      }
    }
     
  # 部署应用到k8s集群
  stage('Deploying to K8s') {
    environment {
      HARBOR_USER = credentials('HARBOR_ACCOUNT')
    }
    steps {
      container(name: 'helm'){
       sh """
         helm repo add --username ${HARBOR_USER_USR} --password ${HARBOR_USER_PSW} ${REGISTRY_DIR}-${ENV} http://${HARBOR_ADDRESS}/chartrepo/${REGISTRY_DIR} \
         && cd chart \
         && helm cm-push ${APP_NAME} ${REGISTRY_DIR}-${ENV} \
         && helm repo update \
         && helm --kubeconfig /mnt/.kube/config upgrade -i ${APP_NAME} ${REGISTRY_DIR}-${ENV}/${APP_NAME} -n ${ENV} \
         --set image.registry=${HARBOR_ADDRESS} \
         --set image.repository=${REGISTRY_DIR}/${APP_NAME} \
         --set image.tag=${TAG} \
         --set replicaCount=1 \
         --set autoscaling.enabled=false \
         --set resources.limits.cpu=1000m,resources.limits.memory=2048Mi,resources.requests.cpu=100m,resources.requests.memory=1024Mi \
         --set env[0].name="CONFIG_SERVER_URI",env[0].value="${CONFIG_SERVER_URI}" \
         --set env[1].name="JAVA_OPTS",env[1].value="-Xms1024m -Xmx1024m -Dfile.encoding=UTF-8 -Dspring.profiles.active=${ENV} -Djava.awt.headless=true" \
         --set env[2].name="APP_PORT",env[2].value="--server.port=8080" \
         --set env[3].name="SW_AGENT_PATH",env[3].value="-javaagent:/home/test/Apps/skywalking/agent/skywalking-agent.jar -Dskywalking.collector.backend_service=${SKYWALKING_SERVICE} -Dapp.id=${APP_NAME} -Dskywalking.agent.service_name=${APP_NAME}"
         """
        }
      }
    }
  }
  # 定义一些全局的环境变量
  environment {
    COMMIT_ID = ""
    HARBOR_ADDRESS = "harbor.test.lo:5000" # Harbor地址
    REGISTRY_DIR = "test"  # Harbor的项目目录,用于存放镜像
    APP_NAME = sh(returnStdout: true, script: 'echo ${JOB_NAME} | awk -F. \'{print \$1}\'').trim()  # 应用名，根据JOB_NAME获取，JOB_NAME名实例appname.dev
    ENV= sh(returnStdout: true, script: 'echo ${JOB_NAME} | awk -F. \'{print \$2}\'').trim()  # 环境和命名空间，根据JOB_NAME获取, JOB_NAME名实例appname.dev
    TAG = ""
    CONFIG_SERVER_URI = "http://nacos-cs.mdw:8848"  # nacos地址
    SKYWALKING_SERVICE = "skywalking-oap.skywalking:11800"  # skywalking地址
  }
  
  parameters {
    # GitParameter插件
    gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE',tagFilter: '*', type:'PT_BRANCH')
  }
}

Dockerfile应用放置于代码仓库的根目录下，具体内容示例如下：

FROM harbor.test.lo:5000/dezhu/openjdk:8u345-b01-jre

RUN groupadd test -g 600 \
    && useradd -m -s /bin/bash test-u 600 -g 600

USER test 

RUN mkdir -p ~/Apps ~/Logs/Java ~/Data ~/Conf

COPY appname/target/*.jar Apps/appname.jar

CMD ["sh", "-c", "source /etc/profile;java ${JAVA_OPTS} ${SW_AGENT_PATH} ${SW_AGENT_COLLECTOR_BACKEND_SERVICES} ${SW_AGENT_NAME} -jar Apps/appname.jar ${APP_PORT}"]

7. 配置pipline流水线工程

登陆Jenkins -> 新建任务 -> 流水线 -> 选择‘Pineline script from SCM’

• Repository URL: http://x.x.x:90/root/x.git
• Credentials: （刚创建的gitlab凭证）
• 指定分支（为空时代表any）: */master (按需修改)
• 脚本路径：Jenkinsfile（和gitlab中文件名对应）
  注：由于Jenkins参数由Jenkinsfile生成，所以第一次执行流水线会失败，第二次构建就能正常使用。

安装Loki日志监控

# 下载helm chart 
helm repo add grafana https://grafana.github.io/helm-charts
helm fetch grafana/loki-stack --version 2.1.2

# 准备离线镜像
grafana/loki:2.0.0
grafana/promtail:2.0.0

# helm安装安装Loki
helm upgrade --install loki ./loki-stack --namespace loki-stack --create-namespace \
--set loki.persistence.enabled=true --set loki.persistence.storageClassName=longhorn \
--set loki.persistence.size=100Gi \
--set loki.service.labels."app\.kubernetes\.io\/name"=loki \
--set loki.image.repository=harbor.dezhu.lo:5000/grafana/loki \
--set promtail.volumes[0].name=docker,promtail.volumes[0].hostPath.path="/home/finance/docker-data/containers" \
--set promtail.volumes[1].name=pods,promtail.volumes[1].hostPath.path="/var/log/pods" \
--set promtail.volumeMounts[0].name=docker,promtail.volumeMounts[0].mountPath="/home/finance/docker-data/containers",promtail.volumeMounts[0].readOnly=true \
--set promtail.volumeMounts[1].name=pods,promtail.volumeMounts[1].mountPath="/var/log/pods",promtail.volumeMounts[1].readOnly=true \
--set promtail.image.repository=harbor.dezhu.lo:5000/grafana/promtail \
--set loki.config.table_manager.retention_deletes_enabled=true \
--set loki.config.table_manager.retention_period=336h

安装kube-prometheus监控系统

# 也可以参考https://www.jianshu.com/p/c01251cf881e安装官方版本并配置HPA自动伸缩容
# 下载chart
helm repo add my-repo https://charts.bitnami.com/bitnami
helm search repo my-repo/kube-prometheus -l
helm fetch my-repo/kube-prometheus --version 8.1.11

# 准备离线镜像
bitnami/prometheus-operator:0.60.1-debian-11-r0
bitnami/prometheus:2.39.1-debian-11-r1
bitnami/thanos:0.28.1-scratch-r0
bitnami/alertmanager:0.24.0-debian-11-r46
bitnami/blackbox-exporter:0.22.0-debian-11-r23
bitnami/node-exporter:1.4.0-debian-11-r2
bitnami/kube-state-metrics:2.6.0-debian-11-r12
bitnami/nginx:1.16.1-debian-10-r63

# helm安装kube-prometheus
helm upgrade --install kube-prometheus ./kube-prometheus --namespace monitoring --create-namespace \
--set global.imageRegistry=harbor.dezhu.lo:5000 --set global.storageClass=longhorn

# 暴露NodePort方式
helm upgrade --install kube-prometheus ./kube-prometheus --namespace monitoring --create-namespace \
--set global.imageRegistry=harbor.dezhu.lo:5000 --set global.storageClass=longhorn \
--set prometheus.service.type=NodePort --set prometheus.service.nodePort=30010 


# 【此版本无需手动添加，因为此版本的prometheus默认使用clusterRole跨namespace访问】kube-prometheus添加自定义监控redis

# mdw 创建Role，此Role赋权限可以读取mdw命名空间内的api
cat< 80
      for: 1m
      labels:
        name: instance
        severity: Warning
      annotations:
        summary: " {{ $labels.alias }}"
        description: " Redis连接数超过最大连接数的80%. "
        value: "{{ $value }}"
    - alert: Redis linked
      expr: redis_connected_clients / redis_config_maxclients * 100 > 80
      for: 1m
      labels:
        name: instance
        severity: Warning
      annotations:
        summary: " {{ $labels.alias }}"
        description: " Redis连接数超过最大连接数的80%. "
        value: "{{ $value }}"
EOF

# 监控外部mysql 
# 需要在监控的目标数据库创建账号用于收集数据库监控指标
CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'pwd4test';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
flush privileges;
# 安装mysqld_exporter
wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.14.0/mysqld_exporter-0.14.0.linux-amd64.tar.gz
sudo tar xf mysqld_exporter-0.14.0.linux-amd64.tar.gz -C /opt
sudo mv /opt/mysqld_exporter-0.14.0.linux-amd64 /opt/mysqld_exporter
# 创建配置文件，按需修改服务器信息
cat <

安装rabbitmq

# 下载chart
helm repo add bitnami https://charts.bitnami.com/bitnami
helm pull bitnami/rabbitmq --version=8.25.0

# 准备离线镜像
bitnami/rabbitmq:3.9.11-debian-10-r28
bitnami/bitnami-shell:10-debian-10-r296
bitnami/nginx:1.16.1-debian-10-r63

# helm安装rabbitmq (注意修改prometheus的namespace)
helm upgrade --install rabbitmq ./rabbitmq --namespace mdw --create-namespace \
--set global.storageClass=longhorn \
--set image.registry=harbor.dezhu.lo:5000 \
--set image.repository=bitnami/rabbitmq \
--set image.tag=3.9.11-debian-10-r28 \
--set replicaCount=3 \
--set metrics.enabled=true \
--set metrics.serviceMonitor.enabled=true \
--set metrics.serviceMonitor.namespace=monitoring \
--set ingress.enabled=true \
--set ingress.hostname=rabbitmq.test.lo \
--set ingress.annotations."kubernetes\.io\/ingress\.class"=nginx \
--set persistence.size=4G \
--set auth.username=dezhu \
--set auth.password=pwd4test \
--set extraEnvVars[0].name=TZ \
--set extraEnvVars[0].value="Asia/Shanghai" \
--set volumePermissions.enabled=true \
--set volumePermissions.image.registry=harbor.dezhu.lo:5000 \
--set volumePermissions.image.repository=bitnami/bitnami-shell \
--set volumePermissions.image.tag=10-debian-10-r296

# 配置本机/etc/hosts，访问验证
xx.xx.xx.xx rabbitmq.test.lo

http:// rabbitmq.test.lo/

安装redis cluster (6.2.6)

# 下载helm chart 
helm pull bitnami/redis-cluster --version=7.5.0

# 准备离线镜像
bitnami/redis-cluster:6.2.6-debian-10-r193
bitnami/bitnami-shell:10-debian-10-r402
bitnami/redis-exporter:1.37.0-debian-10-r33
bitnami/nginx:1.16.1-debian-10-r63 

# helm安装redis-cluster (注意修改prometheus的namespace)
helm upgrade --install redis-cluster ./redis-cluster --namespace mdw --create-namespace \
--set image.registry=harbor.dezhu.lo:5000 \
--set image.repository=bitnami/redis-cluster \
--set image.tag=6.2.6-debian-10-r193 \
--set usePassword=true \
--set password="pwd4test" \
--set persistence.enabled=true \
--set persistence.storageClass=longhorn \
--set persistence.size=2Gi \
--set metrics.enabled=true \
--set metrics.image.registry=harbor.dezhu.lo:5000  \
--set metrics.image.repository=bitnami/redis-exporter \
--set metrics.image.tag=1.37.0-debian-10-r33 \
--set metrics.serviceMonitor.enabled=true \
--set metrics.serviceMonitor.namespace=monitoring

# 验证

redis-cli -c -h redis-cluster -a pwd4test

安装mysql

# 拉取chart
elm pull bitnami/mysql --version 9.4.0

# 准备离线镜像
bitnami/mysql:8.0.30-debian-11-r27
bitnami/bitnami-shell:11-debian-11-r39
bitnami/mysqld-exporter:0.14.0-debian-11-r45
bitnami/nginx:1.16.1-debian-10-r63

 # helm安装mysql (注意修改prometheus的namespace)
helm upgrade --install mysql ./mysql --namespace mdw --create-namespace \
--set global.storageClass=longhorn \
--set global.imageRegistry=harbor.dezhu.lo:5000 \
--set image.repository=bitnami/mysql \
--set image.tag=8.0.30-debian-11-r27 \
--set volumePermissions.image.repository=bitnami/bitnami-shell \
--set volumePermissions.image.tag=11-debian-11-r39 \
--set primary.persistence.size=10Gi \
--set secondary.persistence.size=10Gi \
--set auth.rootPassword=pwd4test \
--set metrics.enabled=true \
--set metrics.image.repository=bitnami/mysqld-exporter \
--set metrics.image.tag=0.14.0-debian-11-r45 \
--set metrics.prometheusRule.enabled=true \
--set metrics.serviceMonitor.enabled=true \
--set metrics.serviceMonitor.namespace=monitoring

# 登陆Mysql验证
mysql -h mysql.mdw.svc.cluster.local -uroot -p

安装grafana

# 下载helm chart 
helm fetch my-repo/grafana --version=8.2.11

# 准备离线镜像
bitnami/grafana:9.1.7-debian-11-r0
bitnami/grafana-image-renderer:3.6.1-debian-11-r10
bitnami/bitnami-shell:11-debian-11-r38

# helm安装grafana
helm upgrade --install grafana ./grafana --namespace monitoring --create-namespace \
--set global.imageRegistry=harbor.dezhu.lo:5000 --set global.storageClass=longhorn \
--set admin.user=admin --set admin.password=pwd4test \
--set service.type=NodePort --set service.nodePorts.grafana=30002

# 登陆grafana验证
http://10.99.73.137:30002/

# 添加kube-prometheus的datasource
setting -> Data source -> 删除已存在的的prometheus
setting -> Data source -> prometheus 
  Name: Prometheus
  URL: http://kube-prometheus-prometheus.monitoring.svc.cluster.local:9090
save & test

# 下载prometheus 的dashboard模板 （https://grafana.com/grafana/dashboards/?search=starsl 搜索关键字starsl）
https://grafana.com/grafana/dashboards/13105
https://grafana.com/grafana/dashboards/13105-1-k8s-for-prometheus-dashboard-20211010/

# 导入dashboard模板
Dashboard -> Browse -> import -> 上传文件 -> load -> 选择VictoriaMetrics:Prometheus -> import 

# 添加Loki的datasource
setting -> Data source -> Loki 
  Name: Loki
  URL: http://loki.loki-stack.svc.cluster.local:3100
save & test

Export -> Data Source: Loki -> Labes：xxx -> Run Query -> 查看日志

# 导入dashboard模板
https://grafana.com/grafana/dashboards/15141


# 导入rabbitmq的dashboard
https://grafana.com/grafana/dashboards/10991