手把手教你使用K8S部署Apollo高可用
一、下载源码
Apollo-git地址
二、创建数据库
按照官网wiki创建数据库
三、修改configservice配置
1. 修改config/apollo-configservice.conf
只修改下面内容，调整容器内日志目录
LOG_FOLDER=~/opt/logs/config/
2. 修改config/application-github.properties 【可改可不改，建议修改防止意外】
增加如下配置，禁用apollo的eureka，本次部署使用独立eureka集群
apollo.eureka.server.enabled=false
3. 修改resources/application.yml
修改端口号，避免端口冲突，修改日志目录为config/apollo-configservice.conf中配置目录，指定文件名
server:
port: 28080
logging:
file: ～/opt/logs/config/apollo-configservice.log
4. 修改bootstrap.yml
按注释修改，其余不用动
eureka:
instance:
# 修改此配置如下 30003端口号在第九步中说明 k8s-node-ip替换为自己k8s节点ip
homePageUrl: http://k8s-node-ip:30003
# 修改此配置如下
preferIpAddress: false
# 增加此属性， 禁用apollo自带的eureka
apollo:
eureka:
server:
enabled: false
5. 修改configservice.properties
只修改这两个属性，与resources/application.yml对应
server.port= 28080
logging.file= ～/opt/logs/config/apollo-configservice.log
6. 修改scripts/startup.sh
只修改这两个属性，与上面对应
## Adjust log dir if necessary
LOG_DIR=~/opt/logs/config
## Adjust server port if necessary
SERVER_PORT=${SERVER_PORT:=28080}
四、修改adminservier配置
1. 修改config/apollo-adminservice.conf
日志路径修改为当前用户目录下opt/logs/admin/文件夹内
LOG_FOLDER=~/opt/logs/portal/
2. 修改config/application-github.properties 【可改可不改，建议修改防止意外】
增加如下配置，禁用apollo的eureka，本次部署使用独立eureka集群
apollo.eureka.server.enabled=false
3. 修改resources/adminservice.properties
修改端口号，避免端口冲突，修改日志目录为config/apollo-adminservice.conf中配置目录，指定文件名
server.port= 28090
logging.file= ~/opt/logs/admin/apollo-adminservice.log
4. 修改resources/application.yml
修改端口号，避免端口冲突，修改日志目录为config/apollo-adminservice.conf中配置目录，指定文件名
server:
port: 28090
logging:
file: ~/opt/logs/admin/apollo-adminservice.log
5. 修改resources/bootstrap.yml
按注释修改，其余不用动
eureka:
instance:
# 修改此配置如下 28091端口号在第十步中说明 k8s-node-ip替换为自己k8s节点ip
homePageUrl: http://k8s-node-ip:28091
# 修改此配置如下
preferIpAddress: false
# 增加此属性， 禁用apollo自带的eureka
apollo:
eureka:
server:
enabled: false
6. 修改scripts/startup.sh
只修改这两个属性，与上面对应
## Adjust log dir if necessary
LOG_DIR=~/opt/logs/admin
## Adjust server port if necessary
SERVER_PORT=${SERVER_PORT:=28090}
五、修改portal配置
1. 修改config/apollo-portal.conf
日志路径修改为当前用户目录下opt/logs/portal/文件夹内
LOG_FOLDER=~/opt/logs/portal/
2. 修改resources/apollo-env.properties【可改可不改，建议修改】
修改这里的端口号为configservice服务的端口号
local.meta=http://localhost:28080
3. 修改resources/application.yml
修改端口号，避免端口冲突，修改日志目录为config/apollo-portalservice.conf中配置目录，指定文件名
server:
port: 28070
logging:
file: ~/opt/logs/portal/apollo-portal.log
4. 修改scripts/startup.sh
## Adjust log dir if necessary
LOG_DIR=~/opt/logs/portal
## Adjust server port if necessary
SERVER_PORT=${SERVER_PORT:=28070}
六、修改Dockerfile内容
1. 修改scripts/apollo-on/kubernetes/apollo-config-server/Dockerfile
只修改下面属性值，这里重新指定k8s中服务命名空间
ENV APOLLO_CONFIG_SERVICE_NAME=“service-apollo-config-server.apollo”
2. 修改scripts/apollo-on/kubernetes/apollo-config-server/scripts/startup-kubernetes.sh
只修改端口号与日志存放目录
## Adjust log dir if necessary
LOG_DIR=~/opt/logs/config
## Adjust server port if necessary
SERVER_PORT=28080
3. 修改scripts/apollo-on/kubernetes/apollo-admin-server/Dockerfile
只修改下面属性值，这里重新指定k8s中服务命名空间
ENV APOLLO_ADMIN_SERVICE_NAME=“service-apollo-admin-server.apollo”
4. 修改scripts/apollo-on/kubernetes/apollo-admin-server/scripts/startup-kubernetes.sh
只修改端口号与日志存放目录
## Adjust log dir if necessary
LOG_DIR=~/opt/logs/apollo-admin-server
## Adjust server port if necessary
SERVER_PORT=28090
5. 修改scripts/apollo-on/kubernetes/apollo-portal-server/Dockerfile
只修改下面属性值，这里重新指定k8s中服务命名空间
ENV APOLLO_PORTAL_SERVICE_NAME=“service-apollo-portal-server.apollo”
6. 修改scripts/apollo-on/kubernetes/apollo-portal-server/scripts/startup-kubernetes.sh
只修改端口号与日志存放目录
## Adjust log dir if necessary
LOG_DIR=~/opt/logs/apollo-portal-server
## Adjust server port if necessary
SERVER_PORT=28070
七、执行编译脚本
在scripts/目录下运行./build.sh
八、编译镜像并push到docker仓库
1. copy并重命名jar包
- 解压apollo-configservice/target/apollo-configservice-1.6.0-SNAPSHOT-github.zip, 将apollo-configservice-1.6.0-SNAPSHOT.jar复制到scripts/apollo-on/kubernetes/apollo-config-server/，并重命名为apollo-configservice.jar
- 解压apollo-adminservice/target/apollo-adminservice-1.6.0-SNAPSHOT-github.zip, 将apollo-adminservice-1.6.0-SNAPSHOT.jar复制到scripts/apollo-on/kubernetes/apollo-admin-server/，并重命名为apollo-adminservice.jar
- 解压apollo-portalservice/target/apollo-portalservice-1.6.0-SNAPSHOT-github.zip, 将apollo-portalservice-1.6.0-SNAPSHOT.jar复制到scripts/apollo-on/kubernetes/apollo-portal-server/，并重命名为apollo-portalservice.jar
2. 编译镜像并push
在以上jar包对应目录下分别执行：
docker build -t imageName:tag .
docker push imageName:tag
注意，build命令后有个点
九、apollo-config-server.yml
# 在pod创建之前先创建configMap
# pod可以通过三种方式使用configMap
# 1. 将configMap中的数据设置为环境变量
# 2. 将configMap中的数据设置为命令行参数
# 3. 使用volume将configMap作为文件或目录挂载
kind: ConfigMap
apiVersion: v1
metadata:
namespace: apollo
name: configmap-apollo-config-server-dev
data:
# 【注意这里数据库连接与eureka连接在不同环境都需要切换】目前由于资源问题只提供dev环境，
# 数据库通过ip连接 【改成自己的数据库连接】
# eureka通过config-server-pod-name-index.meta-server-service-name:prot/eureka/指定 【没通过k8s部署的直接用自己的eureka服务】
application-github.properties: |
spring.datasource.url = jdbc:mysql://mysql-server-ip:3306/ApolloConfigDB_Dev?characterEncoding=utf8
spring.datasource.username = username
spring.datasource.password = password
eureka.service.url = http://eureka-service-0.eureka-service.default:9008/eureka/,http://eureka-service-1.eureka-service.default:9008/eureka/,http://eureka-service-2.eureka-service.default:9008/eureka/
---
kind: Service
apiVersion: v1
metadata:
namespace: apollo
name: service-apollo-meta-server-dev
labels:
app: service-apollo-meta-server-dev
spec:
ports:
- protocol: TCP
port: 28080 # 设定Serivce对外提供服务的端口
targetPort: 28080 # 设定容器(Pod)的端口,即Pod网络的端口
nodePort: 30003 # 将meta-server服务对节点外开发，让开发机器也能访问到k8s中的apollo
selector:
app: pod-apollo-config-server-dev
type: NodePort
# clusterIP: None
sessionAffinity: ClientIP
---
kind: Service
apiVersion: v1
metadata:
namespace: apollo
name: service-apollo-config-server-dev
labels:
app: service-apollo-config-server-dev
spec:
ports:
- protocol: TCP
port: 28080
targetPort: 28080
nodePort: 30002 # 它仅在type为NodePort时才需要指定
selector:
app: pod-apollo-config-server-dev
type: NodePort # NodePort类型的service, 可以让k8s节点外的主机都能访问到
sessionAffinity: ClientIP
# RC、Deployment、DaemonSet都是面向无状态的服务，它们所管理的Pod的IP、名字，启停顺序等都是随机的
# StatefulSet是有状态的集合，管理所有有状态的服务
# StatefulSet本质上是Deployment的一种变体，在v1.9版本中已成为GA版本，它为了解决有状态服务的问题，它所管理的Pod拥有固定的Pod名称，启停顺序
# 在StatefulSet中，Pod名字称为网络标识(hostname)，还必须要用到共享存储
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
namespace: apollo
name: statefulset-apollo-config-server-dev
labels:
app: statefulset-apollo-config-server-dev
spec:
serviceName: service-apollo-meta-server-dev # 声明它属于哪个Headless Service.
replicas: 3
selector:
matchLabels:
app: pod-apollo-config-server-dev # has to match .spec.template.metadata.labels
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-config-server-dev # has to match .spec.selector.matchLabels
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pod-apollo-config-server-dev
topologyKey: kubernetes.io/hostname
volumes:
- name: volume-configmap-apollo-config-server-dev
configMap:
name: configmap-apollo-config-server-dev
items:
- key: application-github.properties
path: application-github.properties
containers:
- image: imageName:tag # 这里需要改成自己打的镜像
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-config-server-dev
ports:
- protocol: TCP
containerPort: 28080
resources:
limits:
memory: 512Mi # 限制内存为512M
volumeMounts:
- name: volume-configmap-apollo-config-server-dev
mountPath: /apollo-config-server/config/application-github.properties
subPath: application-github.properties
env:
- name: APOLLO_CONFIG_SERVICE_NAME
value: "service-apollo-config-server-dev.apollo"
readinessProbe:
tcpSocket:
port: 28080
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 28080
initialDelaySeconds: 120
periodSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
十、apollo-admin-server.yml
# 在pod创建之前先创建configMap
# pod可以通过三种方式使用configMap
# 1. 将configMap中的数据设置为环境变量
# 2. 将configMap中的数据设置为命令行参数
# 3. 使用volume将configMap作为文件或目录挂载
kind: ConfigMap
apiVersion: v1
metadata:
namespace: apollo
name: configmap-apollo-admin-server-dev
data:
# 【注意这里数据库连接与eureka连接在不同环境都需要切换】目前由于资源问题只提供dev环境，
# 数据库通过ip连接 【改成自己的数据库连接】
# eureka通过config-server-pod-name-index.meta-server-service-name:prot/eureka/指定 【没通过k8s部署的直接用自己的eureka服务】
application-github.properties: |
spring.datasource.url = jdbc:mysql://mysql-server-ip:3306/ApolloConfigDB_Dev?characterEncoding=utf8
spring.datasource.username = username
spring.datasource.password = password
eureka.service.url = http://eureka-service-0.eureka-service.default:9008/eureka/,http://eureka-service-1.eureka-service.default:9008/eureka/,http://eureka-service-2.eureka-service.default:9008/eureka/
---
kind: Service
apiVersion: v1
metadata:
namespace: apollo
name: service-apollo-admin-server-dev
labels:
app: service-apollo-admin-server-dev
spec:
ports:
- protocol: TCP
port: 28090 # 设定Serivce对外提供服务的端口
targetPort: 28090 # 设定容器(Pod)的端口,即Pod网络的端口
nodePort: 28091
selector:
app: pod-apollo-admin-server-dev
type: NodePort
# clusterIP: None
sessionAffinity: ClientIP
# RC、Deployment、DaemonSet都是面向无状态的服务，它们所管理的Pod的IP、名字，启停顺序等都是随机的
# StatefulSet是有状态的集合，管理所有有状态的服务
# StatefulSet本质上是Deployment的一种变体，在v1.9版本中已成为GA版本，它为了解决有状态服务的问题，它所管理的Pod拥有固定的Pod名称，启停顺序
# 在StatefulSet中，Pod名字称为网络标识(hostname)，还必须要用到共享存储
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: apollo
name: deployment-apollo-admin-server-dev
labels:
app: deployment-apollo-admin-server-dev
spec:
replicas: 3
selector:
matchLabels:
app: pod-apollo-admin-server-dev # has to match .spec.template.metadata.labels
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-admin-server-dev # has to match .spec.selector.matchLabels
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pod-apollo-admin-server-dev
topologyKey: kubernetes.io/hostname
volumes:
- name: volume-configmap-apollo-admin-server-dev
configMap:
name: configmap-apollo-admin-server-dev
items:
- key: application-github.properties
path: application-github.properties
containers:
- image: imageName:tag
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-admin-server-dev
ports:
- protocol: TCP
containerPort: 28090
resources:
limits:
memory: 512Mi
volumeMounts:
- name: volume-configmap-apollo-admin-server-dev
mountPath: /apollo-admin-server/config/application-github.properties
subPath: application-github.properties
env:
- name: APOLLO_CONFIG_SERVICE_NAME
value: "service-apollo-admin-server-dev.apollo"
readinessProbe:
tcpSocket:
port: 28090
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 28090
initialDelaySeconds: 120
periodSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
十一、apollo-portal-server.yml
---
# configmap for apollo-portal-server
kind: ConfigMap
apiVersion: v1
metadata:
namespace: apollo
name: configmap-apollo-portal-server
data:
# 数据库通过ip连接 【改成自己的数据库连接】
application-github.properties: |
spring.datasource.url = jdbc:mysql://mysql-server-ip:3306/ApolloPortalDB?characterEncoding=utf8
spring.datasource.username = username
spring.datasource.password = password
apollo-env.properties: |
dev.meta=http://statefulset-apollo-config-server-dev-0.service-apollo-meta-server-dev:28080,http://statefulset-apollo-config-server-dev-1.service-apollo-meta-server-dev:28080,http://statefulset-apollo-config-server-dev-2.service-apollo-meta-server-dev:28080
# 资源限制，现在只使用dev环境
# fat.meta=http://statefulset-apollo-config-server-fat-0.service-apollo-meta-server-fat:28080,http://statefulset-apollo-config-server-fat-1.service-apollo-meta-server-fat:28080,http://statefulset-apollo-config-server-fat-2.service-apollo-meta-server-fat:28080
# uat.meta=http://service-apollo-config-server-test-beta.sre:8080
# pro.meta=http://service-apollo-config-server-prod.sre:8080
---
kind: Service
apiVersion: v1
metadata:
namespace: apollo
name: service-apollo-portal-server
labels:
app: service-apollo-portal-server
spec:
ports:
- protocol: TCP
port: 28070
targetPort: 28070
nodePort: 30001
selector:
app: pod-apollo-portal-server
type: NodePort
# portal session 保持
sessionAffinity: ClientIP
# 这里使用Deployment
---
kind: Deployment
apiVersion: apps/v1
metadata:
namespace: apollo
name: deployment-apollo-portal-server
labels:
app: deployment-apollo-portal-server
spec:
# 3 个实例
replicas: 3
selector:
matchLabels:
app: pod-apollo-portal-server
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: pod-apollo-portal-server
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pod-apollo-portal-server
topologyKey: kubernetes.io/hostname
volumes:
- name: volume-configmap-apollo-portal-server
configMap:
name: configmap-apollo-portal-server
items:
- key: application-github.properties
path: application-github.properties
- key: apollo-env.properties
path: apollo-env.properties
containers:
- image: imageName:tag # 更改为你的 docker registry 下的 image
securityContext:
privileged: true
imagePullPolicy: IfNotPresent
name: container-apollo-portal-server
ports:
- protocol: TCP
containerPort: 28070
resources:
limits:
memory: 512Mi
volumeMounts:
- name: volume-configmap-apollo-portal-server
mountPath: /apollo-portal-server/config/application-github.properties
subPath: application-github.properties
- name: volume-configmap-apollo-portal-server
mountPath: /apollo-portal-server/config/apollo-env.properties
subPath: apollo-env.properties
env:
- name: APOLLO_PORTAL_SERVICE_NAME
value: "service-apollo-portal-server.apollo"
readinessProbe:
tcpSocket:
port: 28070
initialDelaySeconds: 10
periodSeconds: 5
livenessProbe:
tcpSocket:
port: 28070
# 120s 内, server 未启动则重启 container
initialDelaySeconds: 120
periodSeconds: 15
dnsPolicy: ClusterFirst
restartPolicy: Always
十二、启动服务
分别启动configserver、adminserver、portalserver【必须严格按照此顺序启动，且在上一个服务启动成功后再启动下一个服务】， 命令如下:
kubectl create -f apollo-config-server.yml
kubectl create -f apollo-admin-server.yml
kubectl create -f apollo-portal-server.yml
启动完成后可运行以下命令检查pod启动情况
kubectl get pods -n apollo
结果如下，所有pod都为ready/running，则部署成功
查看某个pod日志，先进入容器
kubectl exec -ti deployment-apollo-portal-server-59479cfddc-xf4fd /bin/sh -n apollo
再进入～/opt/logs目录tail对应log文件
十三、访问Apollo-portal服务
k8s任意节点ip:30001访问，可在管理员工具-系统信息中查看服务状态
到此，部署成功！