K8S部署Apollo配置中心

K8S部署Apollo配置中心

参考文档: https://github.com/apolloconfig/apollo/tree/v1.8.0

	[K8S部署apollo配置中心](https://www.cnblogs.com/Fengyinyong/p/14903725.html)

	[apollo官网文档](https://www.apolloconfig.com/#/zh/README)

1、错误问题记录

在k8s里面部署时也遇到了同样的一些问题,在此记录下:

  • 提示exec user process caused "no such file or directory"

    • 由于将Windows文件拷贝到CentOS里面导致的换行符不对,解决方案是在CentOS虚机内git clone 当前仓储
  • 构建的docker镜像启动失败

    • 同上一个问题,从Windows拷贝过去的文件权限不正确导致的shell脚本执行失败,解决方案同上或者chmod +x
  • 一个环境的admin server由于健康检查不过导致不断重启

    • 数据库帐号填错了导致无法登录数据库引发的问题

2、介绍

Apollo(阿波罗)是携程框架部门研发的分布式配置中心,能够集中化管理应用不同环境、不同集群的配置,配置修改后能够实时推送到应用端,并且具备规范的权限、流程治理等特性,适用于微服务配置管理场景。

服务端基于Spring Boot和Spring Cloud开发,打包后可以直接运行,不需要额外安装Tomcat等应用容器。

Java客户端不依赖任何框架,能够运行于所有Java运行时环境,同时对Spring/Spring Boot环境也有较好的支持。

.Net客户端不依赖任何框架,能够运行于所有.Net运行时环境。

说明:最近在用K8S部署微服务,而微服务的配置文件众多,需要一个配置中心来处理配置文件。于是采用apollo来作为配置中心。本实例介绍了如何采用K8S部署高可用的apollo集群。

3、环境配置

mysql5.7.39

注意:mysql的版本一定要大于5.7,不然导入数据库sql文件会报错的

因为实验方便,给root远程登录权限(切忌,生产环境,禁用root,单独数据库配置单独账号)

数据库名称 IP 作用 远程账号
DevApolloConfigDB 192.168.102.20 DEV环境的配置 root
ProdApolloConfigDB 192.168.102.20 PRO环境的配置 root
ApolloPortalDB 192.168.102.20 WEB界面管理 root

安装Apollo版本:1.8.0(注意:sql文件和jar版本一定要对应)

(目前官方给4套环境部署安装,我这边采用PRO和DEV环境)

已经构建成功的Kubernetes集群,

[root@bt nginx]# kubectl get node 
NAME       STATUS   ROLES                  AGE     VERSION
master01   Ready    control-plane,master   2d16h   v1.23.0
node01     Ready    <none>                 2d16h   v1.23.0
node02     Ready    <none>                 2d16h   v1.23.0

4、Apollo安装

1、克隆Apollo代码

官方地址:https://github.com/apolloconfig/apollo

在Centos上克隆(因为字符原因,所以要在Linux克隆,千万不要win上下载上传,容易出问题)

git clone https://github.com/apolloconfig/apollo.git

cd apollo 

#查看版本,
git tag
#切换分支
git checkout v1.8.0
#创建新的分支,切换
git checkout -b heian

K8S部署Apollo配置中心_第1张图片

2、初始化Mysql数据库

在这里插入图片描述

[root@bt db]# pwd
/opt/apollo/scripts/apollo-on-kubernetes/db
[root@bt db]# ls
config-db-dev  config-db-prod  config-db-test-alpha  config-db-test-beta  portal-db
[root@bt db]# tree .
.
├── config-db-dev
│?? └── apolloconfigdb.sql
├── config-db-prod
│?? └── apolloconfigdb.sql
├── config-db-test-alpha
│?? └── apolloconfigdb.sql
├── config-db-test-beta
│?? └── apolloconfigdb.sql
└── portal-db
    └── apolloportaldb.sql

大胆执行,数据库名称官方定义过,不会冲突
执行dev环境
 cd config-db-dev/
[root@bt config-db-dev]# mysql -uroot -p < apolloconfigdb.sql 
Enter password: 
执行pro环境
 cd config-db-pro/
[root@bt config-db-dev]# mysql -uroot -p < apolloconfigdb.sql 
Enter password: 

执行portal环境
cd portal-db/
[root@bt config-db-dev]# mysql -uroot -p < apolloportaldb.sql

K8S部署Apollo配置中心_第2张图片

K8S部署Apollo配置中心_第3张图片

3、下载jar版本

官方下载地址:https://github.com/apolloconfig/apollo/releases/tag/v1.8.0

cd /opt/apollo
wget https://github.com/apolloconfig/apollo/releases/download/v1.8.0/apollo-adminservice-1.8.0-github.zip
wget https://github.com/apolloconfig/apollo/releases/download/v1.8.0/apollo-configservice-1.8.0-github.zip
wget https://github.com/apolloconfig/apollo/releases/download/v1.8.0/apollo-portal-1.8.0-github.zip

进行解压,需要其中的jar

​    解压 apollo-portal-1.8.0-github.zip
​  获取 apollo-portal-1.8.0.jar, 重命名为 apollo-portal.jar, 放到 scripts/apollo-on-kubernetes/apollo-portal-server

​    解压 apollo-adminservice-1.8.0-github.zip
​  获取 apollo-adminservice-1.8.0.jar, 重命名为 apollo-adminservice.jar, 放到 scripts/apollo-on-kubernetes/apollo-admin-server

​    解压 apollo-configservice-1.8.0-github.zip
​  获取 apollo-configservice-1.8.0.jar, 重命名为 apollo-configservice.jar, 放到 scripts/apollo-on-kubernetes/apollo-config-se

K8S部署Apollo配置中心_第4张图片

4、构建Docker镜像

1、采用K8S部署apollo时,需要用到多个镜像。这些镜像,需要自己构建

2、如果麻烦,可以去官方镜像 https://hub.docker.com/u/apolloconfig

K8S部署Apollo配置中心_第5张图片

docker pull apolloconfig/apollo-configservice:1.8.0
docker pull apolloconfig/apollo-adminservice:1.8.0
docker pull apolloconfig/apollo-portal:1.8.0
还需要一个alpine-bash-3.8 做初始化镜像操作
docker pull zgadocker/alpine-bash-3.8-image:latest
测试这个不行,就按照下方自己打包

K8S部署Apollo配置中心_第6张图片

alpine-bash-3.8-image

[root@bt apollo-on-kubernetes]# cd alpine-bash-3.8-image
[root@bt alpine-bash-3.8-image]# docker build -t harbor.ownit.top/ownit/alpine-bash:3.8 .
[root@bt alpine-bash-3.8-image]# docker push harbor.ownit.top/ownit/alpine-bash:3.8

apollo-config-server

[root@bt apollo-on-kubernetes]# cd apollo-config-server
[root@bt apollo-config-server]# docker build -t harbor.ownit.top/ownit/apollo-configservice:1.8.0 .
[root@bt apollo-config-server]# docker push harbor.ownit.top/ownit/apollo-configservice:1.8.0

apollo-admin-server

[root@bt apollo-on-kubernetes]# cd apollo-admin-server
[root@bt apollo-admin-server]# docker build -t harbor.ownit.top/ownit/apollo-adminservice:1.8.0 .
[root@bt apollo-admin-server]# docker  push harbor.ownit.top/ownit/apollo-adminservice:1.8.0

apollo-portal-server

[root@bt apollo-on-kubernetes]# cd apollo-portal-server
[root@bt apollo-portal-server]# docker build -t harbor.ownit.top/ownit/apollo-portal:1.8.0 .
[root@bt apollo-portal-server]# docker push harbor.ownit.top/ownit/apollo-portal:1.8.0

验证

K8S部署Apollo配置中心_第7张图片

5、配置Yaml文件

K8S部署Apollo配置中心_第8张图片

环境 文件名称 执行顺序
dev service-apollo-config-server-dev.yaml 1
dev service-apollo-admin-server-dev.yaml 2
pro service-apollo-config-server-prod.yaml 3
pro service-apollo-admin-server-prod.yaml 4
portal service-apollo-portal-server.yaml 5

1、service-apollo-config-server-dev.yaml

---
# configmap for apollo-config-server-dev
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: sre
  name: configmap-apollo-config-server-dev
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://192.168.102.20:3306/DevApolloConfigDB?characterEncoding=utf8&serverTimezone=Asia/Shanghai
    spring.datasource.username = root
    spring.datasource.password = 123456
    eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:8080/eureka/

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-meta-server-dev
  labels:
    app: service-apollo-meta-server-dev
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: pod-apollo-config-server-dev
  type: ClusterIP
  clusterIP: None
  sessionAffinity: ClientIP

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-config-server-dev
  labels:
    app: service-apollo-config-server-dev
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30002
  selector:
    app: pod-apollo-config-server-dev 
  type: NodePort
  sessionAffinity: ClientIP

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  namespace: sre
  name: statefulset-apollo-config-server-dev
  labels:
    app: statefulset-apollo-config-server-dev
spec:
  serviceName: service-apollo-meta-server-dev
  replicas: 1
  selector:
    matchLabels:
      app: pod-apollo-config-server-dev
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-config-server-dev
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-config-server-dev
              topologyKey: kubernetes.io/hostname

      volumes:
        - name: volume-configmap-apollo-config-server-dev
          configMap:
            name: configmap-apollo-config-server-dev
            items:
              - key: application-github.properties
                path: application-github.properties
    
      containers:
        - image: harbor.ownit.top/ownit/apollo-configservice:1.8.0
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-config-server-dev
          ports:
            - protocol: TCP
              containerPort: 8080
        
          volumeMounts:
            - name: volume-configmap-apollo-config-server-dev
              mountPath: /apollo-config-server/config/application-github.properties
              subPath: application-github.properties
        
          env:
            - name: APOLLO_CONFIG_SERVICE_NAME
              value: "service-apollo-config-server-dev.sre"
        
          readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
        
          livenessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds:  120
            periodSeconds: 10
        
      dnsPolicy: ClusterFirst
      restartPolicy: Always

2、service-apollo-admin-server-dev.yaml

---
# configmap for apollo-admin-server-dev
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: sre
  name: configmap-apollo-admin-server-dev
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://192.168.102.20:3306/DevApolloConfigDB?characterEncoding=utf8&serverTimezone=Asia/Shanghai
    spring.datasource.username = root
    spring.datasource.password = 123456
    eureka.service.url = http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:8080/eureka/,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:8080/eureka/

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-admin-server-dev
  labels:
    app: service-apollo-admin-server-dev
spec:
  ports:
    - protocol: TCP
      port: 8090
      targetPort: 8090
  selector:
    app: pod-apollo-admin-server-dev
  type: ClusterIP
  sessionAffinity: ClientIP

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: sre
  name: deployment-apollo-admin-server-dev
  labels:
    app: deployment-apollo-admin-server-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pod-apollo-admin-server-dev
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-admin-server-dev
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-admin-server-dev
              topologyKey: kubernetes.io/hostname
    
      volumes:
        - name: volume-configmap-apollo-admin-server-dev
          configMap:
            name: configmap-apollo-admin-server-dev
            items:
              - key: application-github.properties
                path: application-github.properties
    
      initContainers:
        - image: harbor.ownit.top/ownit/alpine-bash:3.8
          name: check-service-apollo-config-server-dev
          command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-config-server-dev.sre:8080"]
    
      containers:
        - image: harbor.ownit.top/ownit/apollo-adminservice:1.8.0
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-admin-server-dev
          ports:
            - protocol: TCP
              containerPort: 8090
        
          volumeMounts:
            - name: volume-configmap-apollo-admin-server-dev
              mountPath: /apollo-admin-server/config/application-github.properties
              subPath: application-github.properties
        
          env:
            - name: APOLLO_ADMIN_SERVICE_NAME
              value: "service-apollo-admin-server-dev.sre"
        
          readinessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 10
            periodSeconds: 5
        
          livenessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 120
            periodSeconds: 10
        
      dnsPolicy: ClusterFirst
      restartPolicy: Always

3、service-apollo-config-server-prod.yaml


---
# configmap for apollo-config-server-prod
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: sre
  name: configmap-apollo-config-server-prod
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://192.168.102.20:3306/ProdApolloConfigDB?characterEncoding=utf8&serverTimezone=Asia/Shanghai
    spring.datasource.username = root
    spring.datasource.password = 123456
    eureka.service.url = http://statefulset-apollo-config-server-prod-0.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-1.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-2.service-apollo-meta-server-prod:8080/eureka/

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-meta-server-prod
  labels:
    app: service-apollo-meta-server-prod
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: pod-apollo-config-server-prod
  type: ClusterIP
  clusterIP: None
  sessionAffinity: ClientIP

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-config-server-prod
  labels:
    app: service-apollo-config-server-prod
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 30005
  selector:
    app: pod-apollo-config-server-prod
  type: NodePort
  sessionAffinity: ClientIP

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  namespace: sre
  name: statefulset-apollo-config-server-prod
  labels:
    app: statefulset-apollo-config-server-prod
spec:
  serviceName: service-apollo-meta-server-prod
  replicas: 1
  selector:
    matchLabels:
      app: pod-apollo-config-server-prod
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-config-server-prod
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-config-server-prod
              topologyKey: kubernetes.io/hostname
    
      volumes:
        - name: volume-configmap-apollo-config-server-prod
          configMap:
            name: configmap-apollo-config-server-prod
            items:
              - key: application-github.properties
                path: application-github.properties
    
      containers:
        - image: harbor.ownit.top/ownit/apollo-configservice:1.8.0
          securityContext:
            privileged: true
          imagePullPolicy: Always
          name: container-apollo-config-server-prod
          ports:
            - protocol: TCP
              containerPort: 8080

          volumeMounts:
            - name: volume-configmap-apollo-config-server-prod
              mountPath: /apollo-config-server/config/application-github.properties
              subPath: application-github.properties
          env:
            - name: APOLLO_CONFIG_SERVICE_NAME
              value: "service-apollo-config-server-prod.sre"
        
          readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5
        
          livenessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 120
            periodSeconds: 10
        
      dnsPolicy: ClusterFirst
      restartPolicy: Always

3、service-apollo-admin-server-prod.yaml

---
# configmap for apollo-admin-server-prod
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: sre
  name: configmap-apollo-admin-server-prod
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://192.168.102.20:3306/ProdApolloConfigDB?characterEncoding=utf8&serverTimezone=Asia/Shanghai
    spring.datasource.username = root
    spring.datasource.password = 123456
    eureka.service.url = http://statefulset-apollo-config-server-prod-0.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-1.service-apollo-meta-server-prod:8080/eureka/,http://statefulset-apollo-config-server-prod-2.service-apollo-meta-server-prod:8080/eureka/

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-admin-server-prod
  labels:
    app: service-apollo-admin-server-prod
spec:
  ports:
    - protocol: TCP
      port: 8090
      targetPort: 8090
  selector:
    app: pod-apollo-admin-server-prod  
  type: ClusterIP
  sessionAffinity: ClientIP

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: sre
  name: deployment-apollo-admin-server-prod
  labels:
    app: deployment-apollo-admin-server-prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pod-apollo-admin-server-prod
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-admin-server-prod
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-admin-server-prod
              topologyKey: kubernetes.io/hostname
    
      volumes:
        - name: volume-configmap-apollo-admin-server-prod
          configMap:
            name: configmap-apollo-admin-server-prod
            items:
              - key: application-github.properties
                path: application-github.properties
    
      initContainers:
        - image: harbor.ownit.top/ownit/alpine-bash:3.8
          name: check-service-apollo-config-server-prod
          command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 50 --retry-delay 1 --retry-max-time 120 service-apollo-config-server-prod.sre:8080"]
    
      containers:
        - image: harbor.ownit.top/ownit/apollo-adminservice:1.8.0
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-admin-server-prod
          ports:
            - protocol: TCP
              containerPort: 8090
        
          volumeMounts:
            - name: volume-configmap-apollo-admin-server-prod
              mountPath: /apollo-admin-server/config/application-github.properties
              subPath: application-github.properties
        
          env:
            - name: APOLLO_ADMIN_SERVICE_NAME
              value: "service-apollo-admin-server-prod.sre"
        
          readinessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 10
            periodSeconds: 5
        
          livenessProbe:
            tcpSocket:
              port: 8090
            initialDelaySeconds: 120
            periodSeconds: 10

      dnsPolicy: ClusterFirst
      restartPolicy: Always

5、service-apollo-portal-server.yaml

---
# 为外部 mysql 服务设置 service
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-mysql-for-portal-server
  labels:
    app: service-mysql-for-portal-server
spec:
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
  type: ClusterIP
  sessionAffinity: None
---
kind: Endpoints
apiVersion: v1
metadata:
  namespace: sre
  name: service-mysql-for-portal-server
subsets:
  - addresses:
      # 更改为你的 mysql addresses, 例如 1.1.1.1
      - ip: 192.168.102.20
    ports:
      - protocol: TCP
        port: 3306

---
# configmap for apollo-portal-server
kind: ConfigMap
apiVersion: v1
metadata:
  namespace: sre
  name: configmap-apollo-portal-server
data:
  application-github.properties: |
    spring.datasource.url = jdbc:mysql://192.168.102.20:3306/ApolloPortalDB?characterEncoding=utf8
    # mysql username
    spring.datasource.username = root
    # mysql password
    spring.datasource.password = 123456
  apollo-env.properties: |
    dev.meta=http://service-apollo-config-server-dev.sre:8080
    fat.meta=http://service-apollo-config-server-test-alpha.sre:8080
    uat.meta=http://service-apollo-config-server-test-beta.sre:8080
    pro.meta=http://service-apollo-config-server-prod.sre:8080

---
kind: Service
apiVersion: v1
metadata:
  namespace: sre
  name: service-apollo-portal-server
  labels:
    app: service-apollo-portal-server
spec:
  ports:
    - protocol: TCP
      port: 8070
      targetPort: 8070
      nodePort: 30001
  selector:
    app: pod-apollo-portal-server
  type: NodePort
  # portal session 保持
  sessionAffinity: ClientIP

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: sre
  name: deployment-apollo-portal-server
  labels:
    app: deployment-apollo-portal-server
spec:
  # 3 个实例
  replicas: 1
  selector:
    matchLabels:
      app: pod-apollo-portal-server
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: pod-apollo-portal-server
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - pod-apollo-portal-server
              topologyKey: kubernetes.io/hostname
    
      volumes:
        - name: volume-configmap-apollo-portal-server
          configMap:
            name: configmap-apollo-portal-server
            items:
              - key: application-github.properties
                path: application-github.properties
              - key: apollo-env.properties
                path: apollo-env.properties
    
      initContainers:
        # 确保 admin-service 正常提供服务
        - image: harbor.ownit.top/ownit/alpine-bash:3.8
          name: check-service-apollo-admin-server-dev
          command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-dev.sre:8090"]
        # - image: harbor.ownit.top/ownit/alpine-bash:3.8
        #   name: check-service-apollo-admin-server-alpha
        #   command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-test-alpha.sre:8090"]
        # - image: harbor.ownit.top/ownit/alpine-bash:3.8
        #   name: check-service-apollo-admin-server-beta
        #   command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-test-beta.sre:8090"]
        - image: harbor.ownit.top/ownit/alpine-bash:3.8
          name: check-service-apollo-admin-server-prod
          command: ['bash', '-c', "curl --connect-timeout 2 --max-time 5 --retry 60 --retry-delay 1 --retry-max-time 120 service-apollo-admin-server-prod.sre:8090"]  
    
      containers:
        - image: harbor.ownit.top/ownit/apollo-portal:1.8.0    # 更改为你的 docker registry 下的 image
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          name: container-apollo-portal-server
          ports:
            - protocol: TCP
              containerPort: 8070
        
          volumeMounts:
            - name: volume-configmap-apollo-portal-server
              mountPath: /apollo-portal-server/config/application-github.properties
              subPath: application-github.properties
            - name: volume-configmap-apollo-portal-server
              mountPath: /apollo-portal-server/config/apollo-env.properties
              subPath: apollo-env.properties
        
          env:
            - name: APOLLO_PORTAL_SERVICE_NAME
              value: "service-apollo-portal-server.sre"
        
          readinessProbe:
            tcpSocket:
              port: 8070
            initialDelaySeconds: 10
            periodSeconds: 5
        
          livenessProbe:
            tcpSocket:
              port: 8070
            # 120s 内, server 未启动则重启 container
            initialDelaySeconds: 120
            periodSeconds: 15
        
      dnsPolicy: ClusterFirst
      restartPolicy: Always

6、执行Yaml文件

# create namespace
kubectl create namespace sre

dev环境
kubectl apply -f apollo-env-dev/service-apollo-config-server-dev.yaml --record 
kubectl apply -f apollo-env-dev/service-apollo-admin-server-dev.yaml --record

pro环境
kubectl apply -f apollo-env-prod/service-apollo-config-server-prod.yaml --record
kubectl apply -f apollo-env-prod/service-apollo-admin-server-prod.yaml --record

K8S部署Apollo配置中心_第9张图片

7、配置ingress域名

apollo-ingress.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: sre
  name: service-apollo-portal-server
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: "example-issuer"
  labels:
    app: service-apollo-portal-server
spec:
  rules:
  - host: apollo.ownit.top
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: service-apollo-portal-server
            port:
              number: 8070

在这里插入图片描述

8、配置nginx代理

nginx配置

apollo.ownit.top.conf

server {
        listen       80;
        server_name  apollo.ownit.top;
        rewrite ^/(.*)$ https://$host/$1 permanent;
        # IP白名单
        include /opt/nginx/whitelist/corporation.conf;
}
server {
        listen       443 ssl;
        server_name  apollo.ownit.top;
        # IP白名单
        include /opt/nginx/whitelist/corporation.conf;

        #ssl                   on;
        ssl_certificate      /opt/nginx/ssl/ownit.top.crt;
        ssl_certificate_key  /opt/nginx/ssl/ownit.top.key;
        include ssl.conf;

        location / {
            proxy_pass  http://kubernetes-cluster; #转发到k8s的ingress 80端口
            include https_proxy.conf;
        }
	access_log  /www/wwwlogs/apollo.ownit.top.log;
    error_log  /www/wwwlogs/apollo.ownit.top.error.log;

}

[root@bt nginx]# nginx  -t 
nginx: the configuration file /www/server/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /www/server/nginx/conf/nginx.conf test is successful
[root@bt nginx]# nginx  -s reload

kubernetes-cluster.conf 转发

[root@bt nginx]# cat kubernetes-cluster.conf 
upstream kubernetes-cluster {
  server 192.168.102.40 weight=5;
  keepalive 16;
}


9、配置powerdns域名解析

K8S部署Apollo配置中心_第10张图片
K8S部署Apollo配置中心_第11张图片

10、测试验证

https://apollo.ownit.top/

默认账号/密码:apollo / admin

如果没有ingress,则可使用nodeport访问

k8s节点ip+30001 端口即可访问

http://k8s节点ip:30001/

K8S部署Apollo配置中心_第12张图片
K8S部署Apollo配置中心_第13张图片

5、Apollo配置使用

1、添加超级管理员用户

1、管理员用户选择 管理员工具->系统参数

K8S部署Apollo配置中心_第14张图片

2、填入系统内置Key:superAdmin 并点击查询,在value部分添加要加入人的用户id(LDAP用户ID为准,可通过wiki后台用户列表查询)后保存即可

K8S部署Apollo配置中心_第15张图片

2、新增部门:organizations

[{"orgId":"kaifa","orgName":"开发部门"},{"orgId":"yunwei","orgName":"运维部门"},{"orgId":"ceshi","orgName":"测试部门"}]

K8S部署Apollo配置中心_第16张图片

3、新增环境:apollo.portal.envs

apollo.portal.envs

K8S部署Apollo配置中心_第17张图片

6、namespace管理

私有namespace

1、添加namespace

Namespace作为配置的分类,可当成一个配置文件。
以添加rocketmq配置为例,添加"spring-rocketmq”Namespace配置rocketmq相关信息。

1、添加项目私有Namespace:spring-rocketmq
进入项目首页,点击左下脚的”添加Namespace”,共包括两项:关联公共Namespace和创建Namespace ,这里选择"创建Namespace"

K8S部署Apollo配置中心_第18张图片
K8S部署Apollo配置中心_第19张图片

2、填写详细的信息
有两种选择,一种是public(所有项目的);另一种是private(当前项目的);如下图所示

K8S部署Apollo配置中心_第20张图片

K8S部署Apollo配置中心_第21张图片

3、提交过后会自动跳转到下面的页面(操控权限)

K8S部署Apollo配置中心_第22张图片

4、不操作返回首页

K8S部署Apollo配置中心_第23张图片

K8S部署Apollo配置中心_第24张图片

2、为namespace操作配置

可以按之前的方法进行操作,也可以通过下面的内容进行批量操作
如:

rocketmq.name-server = 127.0.0.0:9876
rocketmq.producer.group = PID_ACCOUNT

K8S部署Apollo配置中心_第25张图片

K8S部署Apollo配置中心_第26张图片

K8S部署Apollo配置中心_第27张图片

发布

K8S部署Apollo配置中心_第28张图片

K8S部署Apollo配置中心_第29张图片

3、获取namespace里面的数据

修改代码

	public class GetConfigTest {
	    public static void main(String[] args) {
	        while (true){
	            try {
	                Thread.sleep(1000);
	            } catch (InterruptedException e) {
	                e.printStackTrace();
	            }

	//            Config appConfig = ConfigService.getAppConfig();
	            //读取指定的namespace下的配置信息
	            Config config = ConfigService.getConfig("spring-rocketmq");
	            //获取配置信息,第一个参数:配置的key,第二个参数:默认值
	            String value = config.getProperty("rocketmq.producer.group", null);
	            System.out.printf("现在:%s, sms.enable: %s%n", new Date().toString(),value);
	        }


	    }
	}

运行结果

K8S部署Apollo配置中心_第30张图片

公共配置

1、添加公共的namespace

在项目开发中,有一些配置可能是通用的,我们可以通过把这些通用的配置放到公共的Namespace中,这样其他项目要使用时可以直接添加需要的Namespace

  1. 新建common-template项目

K8S部署Apollo配置中心_第31张图片

2、新建namespace
K8S部署Apollo配置中心_第32张图片
K8S部署Apollo配置中心_第33张图片
K8S部署Apollo配置中心_第34张图片

3、添加配置信息

K8S部署Apollo配置中心_第35张图片

K8S部署Apollo配置中心_第36张图片

2、前往其他的项目去关联第一步的namespace

K8S部署Apollo配置中心_第37张图片

K8S部署Apollo配置中心_第38张图片

K8S部署Apollo配置中心_第39张图片

3、修改公共配置

修改server.servlet.context-path为:/account-service

K8S部署Apollo配置中心_第40张图片

K8S部署Apollo配置中心_第41张图片

K8S部署Apollo配置中心_第42张图片

K8S部署Apollo配置中心_第43张图片

4、使用代码读取配置信息

注意:如果项目不一样的话,就需要去将之前在运行里面的环境进行修改,如下

K8S部署Apollo配置中心_第44张图片

修改代码:

public class GetConfigTest {
    public static void main(String[] args) {
        while (true){
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }

//            Config appConfig = ConfigService.getAppConfig();
            //读取指定的namespace下的配置信息
            Config config = ConfigService.getConfig("TEST1.spring-boot-http");
            //获取配置信息,第一个参数:配置的key,第二个参数:默认值
            String value = config.getProperty("server.servlet.context-path", null);
            System.out.printf("现在:%s, sms.enable: %s%n", new Date().toString(),value);
        }


    }
}

K8S部署Apollo配置中心_第45张图片

你可能感兴趣的:(#,Kubernetes项目实战,#,Kubernetes应用,kubernetes,java,容器)