k8s 部署rocketmq集群
目录
一、前言
二、构建 rockermq镜像
三、构建rocketmq-dashboard镜像
四、rocketmq部署文件
五、rocketmq-dashboard部署文件
六、调整副本数案例
七、附加信息
八、附录
一、前言
由于网上找到的大部分部署方案以及rocketmq-operator都是不能很灵活的调整副本数，毕竟每个副本对应其唯一的配置的文件，不同的broker实例使用的配置文件都是不一样的，都存在差异，一旦副本变化了，就不能很好的和broker 的配置文件内容一 一对应起来。
在这里创建一个基于k8s部署单master以及多master部署rocketmq集群，并且只需要一个broker配置文件，多个broker实例会自动基于该broker配置文件模板，自动生成不同broker实例的broker配置文件，扩容或者伸缩rocketmq集群 nameserver或者broker副本数的时候不需要理会配置文件，仅仅是调整实例的副本即可。
备注：该方案不适合有slave节点的部署方式。
二、构建 rockermq镜像
rocketmq-namesrv和 rocketmq-broker共用同一个镜像，仅仅是启动命令和启动参数不一样，后期可灵活的通过调整启动命令和启动参数来实现不同的效果（比如通过挂载configMap的方式自定义rocketmq的配置文件，而不需要重建rocketmq的镜像。
## docker build -t test/rocketmq:v1 . --no-cache
### 构建rocketmq 镜像
## cat Dockerfile
FROM docker.io/library/openjdk:8u102-jdk AS JDK
LABEL "作者"=fanqietudou [email protected]
RUN rm -vf /etc/localtime \
&& ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
&& echo "Asia/Shanghai" > /etc/timezone \
&& export LANG=zh_CN.UTF-8
RUN curl -k https://mirrors.tuna.tsinghua.edu.cn/apache/rocketmq/4.9.4/rocketmq-all-4.9.4-bin-release.zip \
-o /tmp/rocketmq-all-4.9.4-bin-release.zip \
&& unzip /tmp/rocketmq-all-4.9.4-bin-release.zip -d /tmp/ \
&& mv /tmp/rocketmq-all-4.9.4-bin-release /opt/rocketmq \
&& rm -rf /tmp/*
RUN sed -ir '/-Xmx/c JAVA_OPT=${JAVA_OPT}' /opt/rocketmq/bin/runserver.sh \
&& sed -ir '/-Xmx/c JAVA_OPT=${JAVA_OPT}' /opt/rocketmq/bin/runbroker.sh
## 运行 MQ 应用时候可以通过环境变量设置 jvm 数值，如：JAVA_OPT="-server -Xms2g -Xmx2g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"
ENV ROCKETMQ_HOME=/opt/rocketmq
WORKDIR $ROCKETMQ_HOME三、构建rocketmq-dashboard镜像
rocketmq-dashboard是一个可视化的rocketmq集群运维监控工具。
# cat Dockerfile
# docker build -t test/rocketmq-dashboard:v1 . --no-cache
# docker run -d --name rocketmq-dashboard -e "JAVA_OPTS=-Drocketmq.namesrv.addr=127.0.0.1:9876" -p 8080:8080 -t apacherocketmq/rocketmq-dashboard:latest
#
# 下载地址 https://mirrors.tuna.tsinghua.edu.cn/apache/rocketmq/rocketmq-dashboard/1.0.0/rocketmq-dashboard-1.0.0-source-release.zip
# github 地址 https://github.com/apache/rocketmq-dashboard.git
# git clone https://github.com/apache/rocketmq-dashboard.git
# 下载源码后，通过mvn 编译 得到 rocketmq-dashboard.jar
# 编译命令 mvn clean package -Dmaven.test.skip=true docker:build
# 也可以直接 拉取别人做好的镜像 docker pull apacherocketmq/rocketmq-dashboard:latest
# 也可以直接使用编译后，代码生成的 Dockerfile 文件构建 镜像
# 由于我的环境 无法直接拉取 java:8 基础镜像，所以我 换了一个基础镜像
# FROM java:8
FROM docker.io/library/openjdk:8u102-jdk AS JDK
VOLUME /tmp
ADD rocketmq-dashboard-*.jar rocketmq-dashboard.jar
RUN sh -c 'touch /rocketmq-dashboard.jar'
ENV JAVA_OPTS=""
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -jar /rocketmq-dashboard.jar" ]四、rocketmq部署文件
该部署文件包含了rocketmq-nameserver和rocketmq-broker两部分内容，此部分要重点关注各自的启动命令和启动参数，启动命令和启动参数（command）是部署成功的关键，不同的需求，需要适当调整启动命令和启动参数（这里我只做简单的启动参数做例子）。
## cat rocketmq.yaml
## kubectl apply -f rocketmq.yaml
---
# rocketmq-namesrv 部署文件
apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
generation: 2
labels:
app: rocketmq-namesrv
name: rocketmq-namesrv
# namespace: mq
spec:
podManagementPolicy: OrderedReady
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: rocketmq-namesrv
serviceName: rocketmq-namesrv
template:
metadata:
creationTimestamp: null
labels:
app: rocketmq-namesrv
spec:
containers:
- command:
- bin/mqnamesrv
env:
- name: JAVA_OPT
value: -server -Xms2g -Xmx2g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m
image: test/rocketmq:v1
imagePullPolicy: IfNotPresent
name: rocketmq-namesrv
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
---
# rocketmq-broker 部署文件
apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
labels:
app: rocketmq-broker
name: rocketmq-broker
# namespace: mq
spec:
podManagementPolicy: OrderedReady
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: rocketmq-broker
serviceName: rocketmq-broker
template:
metadata:
creationTimestamp: null
labels:
app: rocketmq-broker
spec:
containers:
- command:
- bin/mqbroker
- --namesrvAddr=rocketmq-namesrv:9876
env:
- name: JAVA_OPT
value: -server -Xms4g -Xmx4g
image: test/rocketmq:v1
imagePullPolicy: IfNotPresent
name: rocketmq-broker
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
---
# rocketmq-namesrv 服务文件
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
app: rocketmq-namesrv
name: rocketmq-namesrv
# namespace: mq
spec:
clusterIP: None
ports:
- port: 9876
protocol: TCP
targetPort: 9876
selector:
app: rocketmq-namesrv
sessionAffinity: None
type: ClusterIP
---
五、rocketmq-dashboard部署文件
前四个步骤已经完成创建rocketmq集群，接下来是部署一个能实现运维监控rocketmq的可视化web应用。
部署rocketmq-dashboard应用时候重点关注部署文件里面的env环境变量参数JAVA_OPTS，该env环境变量（JAVA_OPTS）决定了应用是否能成功连接到 rocketmq-namesrv 服务。
---
# rocketmq-dashboard 部署文件
# cat rocketmq-dashboard.yaml
# kubectl apply -f rocketmq-dashboard.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
labels:
app: rocketmq-dashboard
name: rocketmq-dashboard
# namespace: mq
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: rocketmq-dashboard
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: rocketmq-dashboard
spec:
containers:
- env:
- name: JAVA_OPTS
value: -Drocketmq.namesrv.addr=rocketmq-namesrv:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false
image: test/rocketmq-dashboard:v1
imagePullPolicy: IfNotPresent
name: rocketmq-dashboard
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
---
# rocketmq-dashboard 服务文件
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
app: rocketmq-dashboard
name: rocketmq-dashboard
# namespace: mq
spec:
clusterIP: 10.96.149.156
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
app: rocketmq-dashboard
sessionAffinity: None
type: ClusterIP
---
六、调整副本数案例
## 副本数按照自己需求调整即可
## 1、 调整 rocketmq-namesrv 副本数
kubectl scale sts rocketmq-namesrv --replicas=3
## 2、调整 rocketmq-broker 副本数
kubectl scale sts rocketmq-broker --replicas=3七、附加信息
由于我的环境是使用Ingress方式访问，在这里我就不在写ingress配置了，如果有需要可以将上边的 rocketmq-dashboard得服务 service修改成nodeport形式访问。
rocketmq-dashboard应用默认端口是8080，登录账号和密码都是: admin。
默认情况rocketmq-dashboard的访问地址是： http://ip:8080 ，在k8s环境，rocketmq-dashboard的访问端口取决于nodeport映射的实际端口。
八、附录
该部分内容是我们测试环境完整的部署文件，该部署方法是挂载自定义的可通过configMap 修改的配置文件。（rocketmq-dashboard是通过Ingress提供外网访问的，该Ingress部署内容部分就不展示了）
### cat rocketmq.yaml
### kubectl apply -f rocketmq.yaml
---
###########################################################
### 1、部署 rocketmq-namesrv
###########################################################
---
### rocketmq-namesrv configMap 配置文件 (无配置)
### rocketmq-namesrv 没有必要挂载 配置文件使用 默认的就好了
---
### rocketmq-namesrv 部署文件，已挂载 configMap 配置文件
apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
labels:
app: rocketmq-namesrv
name: rocketmq-namesrv
# namespace: mq
spec:
podManagementPolicy: OrderedReady
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: rocketmq-namesrv
serviceName: rocketmq-namesrv
template:
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2022-10-26T09:27:22+08:00"
creationTimestamp: null
labels:
app: rocketmq-namesrv
spec:
containers:
- command:
- bin/mqnamesrv
image: 172.18.6.97/test/rocketmq:v1
imagePullPolicy: IfNotPresent
name: rocketmq
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
---
### rocketmq-namesrv 服务文件
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
labels:
app: rocketmq-namesrv
name: rocketmq-namesrv
# namespace: mq
spec:
clusterIP: None
ports:
- port: 9876
protocol: TCP
targetPort: 9876
selector:
app: rocketmq-namesrv
sessionAffinity: None
type: ClusterIP
---
###########################################################
### 2、部署 rocketmq-broker
###########################################################
---
### rocketmq-broker configMap 配置文件
apiVersion: v1
data:
broker.conf: |
# 所属集群名字(同一主从下: Master 和 Slave 名称要一致)
brokerClusterName = RocketMQ-cluster
# Broker 名字，注意此处不同的配置文件填写的不一样
#brokerName = broker-a
# 0 表示 Master，> 0 表示 Slave
brokerId = 0
# Broker 对外服务的监听端口, 如果一台机器上启动了多个Broker，则要设置不同的端口号，避免冲突
listenPort = 10911
# nameServer地址，如果nameServer是多台集群的话，就用分号分割，比如
# namesrvAddr=192.168.0.1:9876;192.168.0.2:9876
# 是否允许 Broker 自动创建Topic，建议线下开启，线上关闭
autoCreateTopicEnable = true
# 是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup = true
# 与fileReservedTime参数呼应，表明在几点做消息删除动作，默认值04表示凌晨4点
deleteWhen = 04
# 在磁盘上保存消息的时长，单位是小时，自动删除超时的消息
fileReservedTime = 48
# brokerRole有3种：SYNC_MASTER、ASYNC_MASTER、SLAVE
# 关键词 SYNC 和 ASYNC 表示 Master 和 Slave 之间同步消息的机制
# SYNC 的意思是当 Slave 和 Master 消息同步完成后，再返回发送成功的状态
brokerRole = ASYNC_MASTER
# 刷盘方式 ASYNC_FLUSH 异步刷盘; SYNC_FLUSH 同步刷盘
flushDiskType = ASYNC_FLUSH
# #存储路径
# storePathRootDir = /opt/rocketmq/store
# #commitLog 存储路径
# storePathCommitLog = /opt/rocketmq/store/commitlog
# #消费队列存储路径存储路径
# storePathConsumeQueue = /opt/rocketmq/store/consumequeue
# #消息索引存储路径
# storePathIndex=/opt/rocketmq/store/index
# #checkpoint 文件存储路径
# storeCheckpoint=/opt/rocketmq/store/checkpoint
#限制的消息大小
maxMessageSize=65536
kind: ConfigMap
metadata:
name: rocketmq-broker
# namespace: mq
---
### rocketmq-broker 部署文件 ，已挂载 configMap 作为配置文件
apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
labels:
app: rocketmq-broker
name: rocketmq-broker
# namespace: mq
spec:
podManagementPolicy: OrderedReady
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: rocketmq-broker
serviceName: rocketmq-broker
template:
metadata:
annotations:
cattle.io/timestamp: "2022-10-26T01:46:48Z"
creationTimestamp: null
labels:
app: rocketmq-broker
spec:
affinity: {}
containers:
- command:
- bin/mqbroker
- --configFile=conf/broker.conf
- --namesrvAddr=rocketmq-namesrv:9876
image: 172.18.6.97/test/rocketmq:v1
imagePullPolicy: IfNotPresent
name: rocketmq
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/rocketmq/conf/broker.conf
name: rocketmq-broker-conf
readOnly: true
subPath: broker.conf
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
name: rocketmq-broker
name: rocketmq-broker-conf
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
---
###########################################################
### 3、部署 rocketmq-dashboard
###########################################################
---
### rocketmq-dashboard configMap 文件
apiVersion: v1
data:
users.properties: |2-
# This file supports hot change, any change will be auto-reloaded without Dashboard restarting.
# Format: a user per line, username=password[,N] #N is optional, 0 (Normal User); 1 (Admin)
# Define Admin
admin=admin,1
# Define Users
user1=user1
user2=user2
kind: ConfigMap
metadata:
name: rocketmq-dashboard
# namespace: mq
---
### rocketmq-dashboard 部署文件
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
labels:
app: rocketmq-dashboard
name: rocketmq-dashboard
# namespace: mq
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: rocketmq-dashboard
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: rocketmq-dashboard
spec:
affinity: {}
containers:
- env:
- name: JAVA_OPTS
value: -Drocketmq.namesrv.addr=rocketmq-namesrv:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false
image: 172.18.6.97/test/rocketmq-dashboard:v1
imagePullPolicy: IfNotPresent
name: rocketmq-dashboard
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: tmp/rocketmq-console/data/users.properties
name: users-properties
readOnly: true
subPath: users.properties
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
name: rocketmq-dashboard
name: users-properties
---
### rocketmq-dashboard 服务文件
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
app: rocketmq-dashboard
name: rocketmq-dashboard
# namespace: mq
spec:
clusterIP: 10.96.149.156
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
app: rocketmq-dashboard
sessionAffinity: None
type: ClusterIP
---
