k8s安装rocketmq记录

说明

测试环境是k8s集群，在上边部署一套单节点的rocketmq

nameserver部署

Service和StatefulSet脚本如下

apiVersion: v1
kind: Service
metadata:
  labels:
    app: rocketmqnamesrv
  name: rocketmqnamesrv
spec:
  type: ClusterIP
  ports:
  - port: 9876
    targetPort: 9876
    name: namesrvport
  selector:
    app: rocketmqnamesrv
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmqnamesrv
spec:
  serviceName: rocketmqnamesrv
  replicas: 1
  selector:
    matchLabels:
      app: rocketmqnamesrv
  template:
    metadata:
     labels:
       app: rocketmqnamesrv
    spec:
      containers:
      - name: rocketmqnamesrv
        image: apache/rocketmq:4.9.4
        imagePullPolicy: IfNotPresent
        env:
          - name: TZ
            value: Asia/Shanghai
          - name: JAVA_OPT_EXT
            value: "-Duser.home=/home/rocketmq -Xms512M -Xmx512M -Xmn128m"
        command: ["sh","/home/rocketmq/rocketmq-4.9.4/bin/mqnamesrv"]
        readinessProbe:
          tcpSocket:
            port: 9876
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 20

broker部署

部署broker，如果不挂盘存储数据的话，很简单。
不挂盘的server和StatefulSet配置如下：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: rocketmqbroker
  name: rocketmqbroker
spec:
  # type: NodePort
  type: ClusterIP
  ports:
  - port: 10911
    targetPort: 10911
    name: broker-port
  selector:
    app: rocketmqbroker
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmqbroker
spec:
  serviceName: rocketmqbroker
  replicas: 1
  selector:
    matchLabels:
      app: rocketmqbroker
  template:
    metadata:
      labels:
        app: rocketmqbroker
    spec:
      containers:
      - name: rocketmqbroker
        image: apache/rocketmq:4.9.4
        env:
          - name: TZ
            value: Asia/Shanghai
          - name: JAVA_OPT_EXT
            value: "-Xms512M -Xmx512M -Xmn128m"
          - name: NAMESRV_ADDR
            value: rocketmqnamesrv:9876
        command: ["sh","/home/rocketmq/rocketmq-4.9.4/bin/mqbroker","-n","rocketmqnamesrv:9876"]
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 10911
      imagePullSecrets:
      - name: registry-pull-secret

不挂盘，重启后数据会丢失，因此还是想挂盘存储数据，中间碰到了一些问题，简单记录如下：
由于docker镜像用的是公共镜像，是rocketmq:rocketmq启动的,volumeMounts挂载的硬盘没写权限，因此启动失败。可通过 fsGroup（指定卷的附加用户组 ID） 解决

spec:
      securityContext:
        fsGroup: 3000
      containers:

其中id需要进入控制台通过命令 id 查看，如下

[rocketmq@rocketmqnamesrv-0 bin]$ id
uid=3000(rocketmq) gid=3000(rocketmq) groups=3000(rocketmq)

最终挂盘的脚本如下：

 apiVersion: v1
kind: Service
metadata:
  labels:
    app: mqbroker
  name: mqbroker
spec:
  # type: NodePort
  type: ClusterIP
  ports:
  - port: 10911
    targetPort: 10911
    name: broker-port
  selector:
    app: mqbroker

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmqbroker
spec:
  serviceName: rocketmqbroker
  replicas: 1
  selector:
    matchLabels:
      app: rocketmqbroker
  template:
    metadata:
      labels:
        app: rocketmqbroker
    spec:
      securityContext:
        fsGroup: 3000
      containers:
      - name: rocketmqbroker
        image: apache/rocketmq:4.9.4
        env:
          - name: TZ
            value: Asia/Shanghai
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: JAVA_OPT_EXT
            value: "-Xms512M -Xmx512M -Xmn128m -XX:ParallelGCThreads=2"
          - name: NAMESRV_ADDR
            value: rocketmqnamesrv:9876
        command: ["sh","/home/rocketmq/rocketmq-4.9.4/bin/mqbroker","-n","rocketmqnamesrv:9876","-c","/home/rocketmq/rocketmq-4.9.4/conf/broker.conf"]
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 10911
        volumeMounts:
        - mountPath: /home/rocketmq/rocketmq-4.9.4/conf/broker.conf
          name: rocketmqbroker-config
          subPath: broker.conf
        - mountPath: /home/rocketmq/data
          name: rocketmqbroker-data
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
          requests:
            cpu: "1"
            memory: 2Gi
      imagePullSecrets:
      - name: registry-pull-secret
      volumes:
      - configMap:
          defaultMode: 420
          name: rocketmqbroker
        name: rocketmqbroker-config
  volumeClaimTemplates:
          - apiVersion: v1
            kind: PersistentVolumeClaim
            metadata:
              name: rocketmqbroker-data
            spec:
              accessModes:
              - ReadWriteMany
              resources:
                requests:
                  storage: 100Gi
              storageClassName: alicloud-disk-ssd
              volumeMode: Filesystem


 

rocketmq-dashboard部署

脚本如下

apiVersion: v1
kind: Service
metadata:
  labels:
    app: rocketmq-dashboard
  name: rocketmq-dashboard
spec:
  type: NodePort
  ports:
  - port: 8080
    targetPort: 8080
    name: port
    nodePort: 30809
  selector:
    app: rocketmq-dashboard
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rocketmq-dashboard
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq-dashboard
  template:
    metadata:
     labels:
       app: rocketmq-dashboard
    spec:
      containers:
      - name: rocketmq-dashboard
        image: apacherocketmq/rocketmq-dashboard
        imagePullPolicy: IfNotPresent
        env:
          - name: TZ
            value: Asia/Shanghai
          - name: JAVA_OPTS
            value: "-Drocketmq.namesrv.addr=rocketmqnamesrv:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false"
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 15

验证

进入broker控制台的rocketmq的bin目录（进入控制台后默认就在bin目录）
执行如下命令。
生产消息（会生成1000条消息，主题是）：

./tools.sh org.apache.rocketmq.example.quickstart.Producer

成功显示内容如下：

SendResult [sendStatus=SEND_OK, msgId=7F00000100F07852E9228B53DEAF03E6, offsetMsgId=AC12050900002A9F00000000000BB4C8, messageQueue=MessageQueue [topic=TopicTest, brokerName=broker-a, queueId=0], queueOffset=999]
SendResult [sendStatus=SEND_OK, msgId=7F00000100F07852E9228B53DEAF03E7, offsetMsgId=AC12050900002A9F00000000000BB588, messageQueue=MessageQueue [topic=TopicTest, brokerName=broker-a, queueId=1], queueOffset=999]
01:18:49.529 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[:9876] result: true
01:18:49.533 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[:10911] result: true
[rocketmq@rocketmqbroker-0 bin]$ 

消费消息：

./tools.sh org.apache.rocketmq.example.quickstart.Consumer

成功显示内容如下：

ConsumeMessageThread_please_rename_unique_group_name_4_1 Receive New Messages: [MessageExt [brokerName=broker-a, queueId=0, storeSize=192, queueOffset=985, sysFlag=0, bornTimestamp=1658971129478, bornHost=/:57834, storeTimestamp=1658971129478, storeHost=/:10911, msgId=AC12050900002A9F00000000000B8AC8, commitLogOffset=756424, bodyCRC=74782942, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message{topic='TopicTest', flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=1000, CONSUME_START_TIME=1658971252955, UNIQ_KEY=7F00000100F07852E9228B53DE8603AE, CLUSTER=DefaultCluster, TAGS=TagA}, body=[72, 101, 108, 108, 111, 32, 82, 111, 99, 107, 101, 116, 77, 81, 32, 57, 52, 50], transactionId='null'}]] 
ConsumeMessageThread_please_rename_unique_group_name_4_6 Receive New Messages: [MessageExt [brokerName=broker-a, queueId=0, storeSize=192, queueOffset=984, sysFlag=0, bornTimestamp=1658971129475, bornHost=/:57834, storeTimestamp=1658971129475, storeHost=/:10911, msgId=AC12050900002A9F00000000000B87C8, commitLogOffset=755656, bodyCRC=736192263, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message{topic='TopicTest', flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=1000, CONSUME_START_TIME=1658971252955, UNIQ_KEY=7F00000100F07852E9228B53DE8303AA, CLUSTER=DefaultCluster, TAGS=TagA}, body=[72, 101, 108, 108, 111, 32, 82, 111, 99, 107, 101, 116, 77, 81, 32, 57, 51, 56], transactionId='null'}]] 

控制台界面

其他问题

rocketmq启动脚本中有计算内存的逻辑，最终java的启动参数会设置内存为
-server -Xms8192M -Xmx8192M -Xmn1200M ，比分配给容器的内存大的多。虽然通过 JAVA_OPTS 重新设置了，但是挺不爽的。
还有就是brokerName，想用POD_NAME作为brokerName，配置文件中不支持${POD_NAME}