1、集群部署概述
1.1、RocketMQ部署结构图:
各部分说明:
(1)Producer:生产者
(2)Consumer:消费者
(3)NameServer:RockerMQ的注册中心,多个实例组成集群,但相互独立,没有信息交换。每个Broker都向所有NameServer实例注册。
(4)Broker:负责存储消息、转发消息。
1.2、RockerMQ集群部署方式
NameServer集群部署简单,多实例之间独立部署。主要是Broker集群部署相对复杂,其部署方式可分成如下:
(1)单Master模式
(2)多Master模式
(3)多Master多Slave模式-同步复制
(4)多Master多Slave模式-异步复制
(5)Dledger高可用集群
本篇文章就是简单介绍下“多Master多Slave模式-同步复制”搭建过程。计划在3台主机上搭建3主3从集群。
2、准备
2.1、RocketMQ下载
从官网上下载:https://rocketmq.apache.org/
版本:rocketmq-all-4.8.0-bin-release.zip
2.2、主机
三台Centos7主机
主机名 | 主机ip | 角色 |
---|---|---|
learn1 | 192.168.31.200 | nameserver、broker-a-master、broker-b-slave |
learn2 | 192.168.31.201 | nameserver、broker-b-master、broker-c-slave |
learn3 | 192.168.31.202 | nameserver、broker-c-master、broker-a-slave |
2.3、jdk
jdk8
3、安装rocketmq
三台主机都要安装
3.1、安装rocketmq
上传rocketmq到主机上/home/wentian/rocketmq/3m3ssync,并解压改名成rocketmq
$ cd /home/wentian/rocketmq/3m3ssync
$ unzip rocketmq-all-4.8.0-bin-release.zip
$ mv rocketmq-all-4.8.0-bin-release rocketmq
3.2、创建目录
$ cd /home/wentian/rocketmq/3m3ssync/rocketmq
$ mkdir logs
$ mkdir sh
$ mkdir data
logs --- 日志目录
sh --- 脚本目录
data --- 数据目录
3.3、修改日志路径
(1)修改nameserver日志文件
日志文件路径:/home/wentian/rocketmq/3m3ssync/rocketmq/conf/logback_namesrv.xml
修改其中所有的日志路径,示例:
原:
${user.home}/logs/rocketmqlogs/namesrv_default.log
改成:
${LOG_HOME}/${TARGET}/namesrv_default.log
LOG_HOME --- 日志根路径
TARGET --- 自定义进程名,这是为了分开同一台主机上不同rocketmq进程日志,便于查看
这两个属性会在启动脚本中配置,见后续启动脚本配置
(2)修改broker日志文件
位置文件路径:/home/wentian/rocketmq/3m3ssync/rocketmq/conf/logback_broker.xml
修改日志路径,同nameserver一样配置
3.4、修改JVM内存参数
(1)修改nameserver节点JVM参数
修改启动JVM内存
修改/home/wentian/rocketmq/3m3ssync/rocketmq/bin/runserver.sh
原:
JAVA_OPT="${JAVA_OPT} -server -Xms4g -Xmx4g -Xmn2g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"
改成:
JAVA_OPT="${JAVA_OPT} -server -Xms512m -Xmx512m -Xmn256m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"
(2)修改broker节点JVM参数
修改JVM内存
修改/home/wentian/rocketmq/3m3ssync/rocketmq/bin/runbroker.sh
原:
JAVA_OPT="${JAVA_OPT} -server -Xms8g -Xmx8g -Xmn4g"
改成:
JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"
4、部署nameserver
三台主机上都部署nameserver相关
4.1、启停脚本
(1)启动脚本
路径:/home/wentian/rocketmq/3m3ssync/rocketmq/sh/startnamesrv.sh
export JAVA_HOME=/home/wentian/app/java/jdk8
export WORK_HOME=/home/wentian/rocketmq/3m3ssync/rocketmq
export LOG_HOME=${WORK_HOME}/logs
export TARGET=namesrv
cd ${WORK_HOME}
nohup sh bin/mqnamesrv > ./logs/namesrv.log 2>&1 &
(2)停止脚本
路径:/home/wentian/rocketmq/3m3ssync/rocketmq/sh/stopnamesrv.sh
export JAVA_HOME=/home/wentian/app/java/jdk8
export WORK_HOME=/home/wentian/rocketmq/3m3ssync/rocketmq
export LOG_HOME=${WORK_HOME}/logs
export TARGET=namesrv
cd ${WORK_HOME}
sh bin/mqshutdown namesrv
4.2、启动namesrv
执行startnamesrv.sh
5、部署broker
5.1、部署第一台192.168.31.200
需要部署两个broker:broker-a-master、broker-b-slave
(1)配置broker-a配置文件
配置文件路径:/home/wentian/rocketmq/3m3ssync/rocketmq/conf/3m-3s-sync/broker-a.properties
#所属集群名称
brokerClusterName=rocketmq-cluster
#broker名字
brokerName=broker-a
#0>表示master,>0表示slave
brokerId=0
#删除文件时间点,默认凌晨4点
deleteWhen=04
#文件保留时间,默认48小时
fileReservedTime=48
#broker角色,ASYNC_MASTER=异步复制Master,SYNC_MASTER=同步双写Master,SLAVE=slave节点
brokerRole=SYNC_MASTER
#刷盘方式,ASYNC_FLUSH=异步刷盘,SYNC_FLUSH=同步刷盘
flushDiskType=ASYNC_FLUSH
#broker对我服务的监听端口,broker会占用三个端口,listenPort-2、listenPort、listenPort+1
listenPort=10910
#nameServer地址,分号分隔多个
namesrvAddr=192.168.31.200:9876;192.168.31.201:9876;192.168.31.202:9876
#默认的Topic队列数,默认是4
defaultTopicQueueNums=4
#自动创建Topic,建议线上关闭,线下开启
autoCreateTopicEnable=true
#自动创建订阅组,建议线上关闭,线下开启
autoCreateSubscriptionGroup=true
#commitLog每个文件的大小默认1G
#mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条,根据业务情况调整
#mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#物理磁盘最大利用百分比,超过则报警
#diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a
#commitLog 存储路径
storePathCommitLog=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a/commitlog
#消费队列存储路径存储路径
storePathConsumerQueue=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a/consumequeue
#消息索引存储路径
storePathIndex=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a/index
#checkpoint 文件存储路径
storeCheckpoint=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a/checkpoint
#abort 文件存储路径
abortFile=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a/abort
注:
a)相关存储路径,在首次启动时会自动创建,主从两台主机就配置相同路径
b)注意不同点配置
brokerName:主从两台主机,配置相同
brokerId:主从两台主机,主配置0,从配置>0,这里配置1,(如果多个从,则依次递增)
brokerRole:主配置SYNC_MASTER,从配置SLAVE
listenPort:监听端口,但是broker启动会在此基础上供监听三个端口,分别是:listenPort-2、listenPort、listenPort+1。
(2)配置broker-b-s.properties配置文件
配置文件路径:/home/wentian/rocketmq/3m3ssync/rocketmq/conf/3m-3s-sync/broker-b-s.properties
#所属集群名称
brokerClusterName=rocketmq-cluster
#broker名字
brokerName=broker-b
#0>表示master,>0表示slave
brokerId=1
#删除文件时间点,默认凌晨4点
deleteWhen=04
#文件保留时间,默认48小时
fileReservedTime=48
#broker角色,ASYNC_MASTER=异步复制Master,SYNC_MASTER=同步双写Master,SLAVE=slave节点
brokerRole=SLAVE
#刷盘方式,ASYNC_FLUSH=异步刷盘,SYNC_FLUSH=同步刷盘
flushDiskType=ASYNC_FLUSH
#broker对我服务的监听端口
listenPort=10920
#nameServer地址,分号分隔多个
namesrvAddr=192.168.31.200:9876;192.168.31.201:9876;192.168.31.202:9876
#默认的Topic队列数,默认是4
defaultTopicQueueNums=4
#自动创建Topic,建议线上关闭,线下开启
autoCreateTopicEnable=true
#自动创建订阅组,建议线上关闭,线下开启
autoCreateSubscriptionGroup=true
#commitLog每个文件的大小默认1G
#mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条,根据业务情况调整
#mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#物理磁盘最大利用百分比,超过则报警
#diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b
#commitLog 存储路径
storePathCommitLog=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b/commitlog
#消费队列存储路径存储路径
storePathConsumerQueue=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b/consumequeue
#消息索引存储路径
storePathIndex=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b/index
#checkpoint 文件存储路径
storeCheckpoint=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b/checkpoint
#abort 文件存储路径
abortFile=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b/abort
(3)broker-a启动脚本
路径:/home/wentian/rocketmq/3m3ssync/rocketmq/sh/startbroker-a.sh
export JAVA_HOME=/home/wentian/app/java/jdk8
export WORK_HOME=/home/wentian/rocketmq/3m3ssync/rocketmq
export LOG_HOME=${WORK_HOME}/logs
export TARGET=broker_a
cd ${WORK_HOME}
nohup sh bin/mqbroker -c conf/3m-3s-sync/broker-a.properties > ./logs/broker-a.log 2>&1 &
(4)broker-b-s启动脚本
路径:/home/wentian/rocketmq/3m3ssync/rocketmq/sh/startbroker-b-s.sh
export JAVA_HOME=/home/wentian/app/java/jdk8
export WORK_HOME=/home/wentian/rocketmq/3m3ssync/rocketmq
export LOG_HOME=${WORK_HOME}/logs
export TARGET=broker_b_s
cd ${WORK_HOME}
nohup sh bin/mqbroker -c conf/3m-3s-sync/broker-b-s.properties > ./logs/broker-b.log 2>&1 &
(5)停止脚本
路径:/home/wentian/rocketmq/3m3ssync/rocketmq/sh/stopbroker.sh
该停止脚本会停掉该主机上所有broker
export JAVA_HOME=/home/wentian/app/java/jdk8
cd /home/wentian/rocketmq/3m3ssync/rocketmq
sh bin/mqshutdown broker
5.2、部署第二台192.168.31.201
部署两个broker:broker-b-master、broker-c-slave
(1)配置broker-b.properties
配置文件路径:/home/wentian/rocketmq/3m3ssync/rocketmq/conf/3m-3s-sync/broker-b.properties
#所属集群名称
brokerClusterName=rocketmq-cluster
#broker名字
brokerName=broker-b
#0>表示master,>0表示slave
brokerId=0
#删除文件时间点,默认凌晨4点
deleteWhen=04
#文件保留时间,默认48小时
fileReservedTime=48
#broker角色,ASYNC_MASTER=异步复制Master,SYNC_MASTER=同步双写Master,SLAVE=slave节点
brokerRole=SYNC_MASTER
#刷盘方式,ASYNC_FLUSH=异步刷盘,SYNC_FLUSH=同步刷盘
flushDiskType=ASYNC_FLUSH
#broker对我服务的监听端口
listenPort=10920
#nameServer地址,分号分隔多个
namesrvAddr=192.168.31.200:9876;192.168.31.201:9876;192.168.31.202:9876
#默认的Topic队列数,默认是4
defaultTopicQueueNums=4
#自动创建Topic,建议线上关闭,线下开启
autoCreateTopicEnable=true
#自动创建订阅组,建议线上关闭,线下开启
autoCreateSubscriptionGroup=true
#commitLog每个文件的大小默认1G
#mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条,根据业务情况调整
#mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#物理磁盘最大利用百分比,超过则报警
#diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b
#commitLog 存储路径
storePathCommitLog=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b/commitlog
#消费队列存储路径存储路径
storePathConsumerQueue=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b/consumequeue
#消息索引存储路径
storePathIndex=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b/index
#checkpoint 文件存储路径
storeCheckpoint=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b/checkpoint
#abort 文件存储路径
abortFile=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-b/abort
(2)配置broker-c-s.properties
配置文件路径:/home/wentian/rocketmq/3m3ssync/rocketmq/conf/3m-3s-sync/broker-c-s.properties
#所属集群名称
brokerClusterName=rocketmq-cluster
#broker名字
brokerName=broker-c
#0>表示master,>0表示slave
brokerId=1
#删除文件时间点,默认凌晨4点
deleteWhen=04
#文件保留时间,默认48小时
fileReservedTime=48
#broker角色,ASYNC_MASTER=异步复制Master,SYNC_MASTER=同步双写Master,SLAVE=slave节点
brokerRole=SLAVE
#刷盘方式,ASYNC_FLUSH=异步刷盘,SYNC_FLUSH=同步刷盘
flushDiskType=ASYNC_FLUSH
#broker对我服务的监听端口
listenPort=10930
#nameServer地址,分号分隔多个
namesrvAddr=192.168.31.200:9876;192.168.31.201:9876;192.168.31.202:9876
#默认的Topic队列数,默认是4
defaultTopicQueueNums=4
#自动创建Topic,建议线上关闭,线下开启
autoCreateTopicEnable=true
#自动创建订阅组,建议线上关闭,线下开启
autoCreateSubscriptionGroup=true
#commitLog每个文件的大小默认1G
#mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条,根据业务情况调整
#mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#物理磁盘最大利用百分比,超过则报警
#diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c
#commitLog 存储路径
storePathCommitLog=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c/commitlog
#消费队列存储路径存储路径
storePathConsumerQueue=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c/consumequeue
#消息索引存储路径
storePathIndex=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c/index
#checkpoint 文件存储路径
storeCheckpoint=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c/checkpoint
#abort 文件存储路径
abortFile=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c/abort
(3)brokder-b启动脚本
脚本路径:/home/wentian/rocketmq/3m3ssync/rocketmq/sh/startbroker-b.sh
export JAVA_HOME=/home/wentian/app/java/jdk8
export WORK_HOME=/home/wentian/rocketmq/3m3ssync/rocketmq
export LOG_HOME=${WORK_HOME}/logs
export TARGET=broker_b
cd ${WORK_HOME}
nohup sh bin/mqbroker -c conf/3m-3s-sync/broker-b.properties > ./logs/broker-b.log 2>&1 &
(4)broker-c-s启动脚本
脚本路径:/home/wentian/rocketmq/3m3ssync/rocketmq/sh/startbroker-c-s.sh
export JAVA_HOME=/home/wentian/app/java/jdk8
export WORK_HOME=/home/wentian/rocketmq/3m3ssync/rocketmq
export LOG_HOME=${WORK_HOME}/logs
export TARGET=broker_c_s
cd ${WORK_HOME}
nohup sh bin/mqbroker -c conf/3m-3s-sync/broker-c-s.properties > ./logs/broker-c.log 2>&1 &
(5)broker停止脚本
同主机1
5.3、部署第三台192.168.31.202
部署两个broker:broker-c-master、broker-a-slave
(1)配置broker-c.properties
配置文件路径:/home/wentian/rocketmq/3m3ssync/rocketmq/conf/3m-3s-sync/broker-c.properties
#所属集群名称
brokerClusterName=rocketmq-cluster
#broker名字
brokerName=broker-c
#0>表示master,>0表示slave
brokerId=0
#删除文件时间点,默认凌晨4点
deleteWhen=04
#文件保留时间,默认48小时
fileReservedTime=48
#broker角色,ASYNC_MASTER=异步复制Master,SYNC_MASTER=同步双写Master,SLAVE=slave节点
brokerRole=SYNC_MASTER
#刷盘方式,ASYNC_FLUSH=异步刷盘,SYNC_FLUSH=同步刷盘
flushDiskType=ASYNC_FLUSH
#broker对我服务的监听端口
listenPort=10930
#nameServer地址,分号分隔多个
namesrvAddr=192.168.31.200:9876;192.168.31.201:9876;192.168.31.202:9876
#默认的Topic队列数,默认是4
defaultTopicQueueNums=4
#自动创建Topic,建议线上关闭,线下开启
autoCreateTopicEnable=true
#自动创建订阅组,建议线上关闭,线下开启
autoCreateSubscriptionGroup=true
#commitLog每个文件的大小默认1G
#mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条,根据业务情况调整
#mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#物理磁盘最大利用百分比,超过则报警
#diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c
#commitLog 存储路径
storePathCommitLog=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c/commitlog
#消费队列存储路径存储路径
storePathConsumerQueue=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c/consumequeue
#消息索引存储路径
storePathIndex=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c/index
#checkpoint 文件存储路径
storeCheckpoint=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c/checkpoint
#abort 文件存储路径
abortFile=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-c/abort
(2)配置broker-a-s.properties
配置文件路径:/home/wentian/rocketmq/3m3ssync/rocketmq/conf/3m-3s-sync/broker-a-s.properties
#所属集群名称
brokerClusterName=rocketmq-cluster
#broker名字
brokerName=broker-a
#0>表示master,>0表示slave
brokerId=1
#删除文件时间点,默认凌晨4点
deleteWhen=04
#文件保留时间,默认48小时
fileReservedTime=48
#broker角色,ASYNC_MASTER=异步复制Master,SYNC_MASTER=同步双写Master,SLAVE=slave节点
brokerRole=SLAVE
#刷盘方式,ASYNC_FLUSH=异步刷盘,SYNC_FLUSH=同步刷盘
flushDiskType=ASYNC_FLUSH
#broker对我服务的监听端口
listenPort=10910
#nameServer地址,分号分隔多个
namesrvAddr=192.168.31.200:9876;192.168.31.201:9876;192.168.31.202:9876
#默认的Topic队列数,默认是4
defaultTopicQueueNums=4
#自动创建Topic,建议线上关闭,线下开启
autoCreateTopicEnable=true
#自动创建订阅组,建议线上关闭,线下开启
autoCreateSubscriptionGroup=true
#commitLog每个文件的大小默认1G
#mapedFileSizeCommitLog=1073741824
#ConsumeQueue每个文件默认存30W条,根据业务情况调整
#mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#物理磁盘最大利用百分比,超过则报警
#diskMaxUsedSpaceRatio=88
#存储路径
storePathRootDir=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a
#commitLog 存储路径
storePathCommitLog=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a/commitlog
#消费队列存储路径存储路径
storePathConsumerQueue=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a/consumequeue
#消息索引存储路径
storePathIndex=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a/index
#checkpoint 文件存储路径
storeCheckpoint=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a/checkpoint
#abort 文件存储路径
abortFile=/home/wentian/rocketmq/3m3ssync/rocketmq/data/store-a/abort
(3)brokder-c启动脚本
脚本路径:/home/wentian/rocketmq/3m3ssync/rocketmq/sh/startbroker-c.sh
export JAVA_HOME=/home/wentian/app/java/jdk8
export WORK_HOME=/home/wentian/rocketmq/3m3ssync/rocketmq
export LOG_HOME=${WORK_HOME}/logs
export TARGET=broker_c
cd ${WORK_HOME}
nohup sh bin/mqbroker -c conf/3m-3s-sync/broker-c.properties > ./logs/broker-c.log 2>&1 &
(4)broker-a-s启动脚本
脚本路径:/home/wentian/rocketmq/3m3ssync/rocketmq/sh/startbroker-a-s.sh
export JAVA_HOME=/home/wentian/app/java/jdk8
export WORK_HOME=/home/wentian/rocketmq/3m3ssync/rocketmq
export LOG_HOME=${WORK_HOME}/logs
export TARGET=broker_a_s
cd ${WORK_HOME}
nohup sh bin/mqbroker -c conf/3m-3s-sync/broker-a-s.properties > ./logs/broker-a.log 2>&1 &
(5)broker停止脚本
同主机1
至此,broker三主三从部署完成,开始启动验证。
6、启动broker
按以下顺序依次启动
(1)启动三台nameserver
(2)启动三台broker master
(3)启动三台broker slave
7、安装rocketmq可视化
7.1、下载
源码下载地址: [https://github.com/apache/rocketmq-externals/tree/master/](https://github.com/apache/rocketmq-externals/tree/master/)
7.2、编译成jar
mvn clean package -Dmaven.test.skip=true
我这边是在本机通过idea编译
7.3、上传jar到任一服务器上,执行
上传到第一台主机192.168.31.200
启动:
java -jar rocketmq-console-ng-2.0.0.jar --server.port=8080 --rocketmq.config.namesrvAddr=192.168.31.200:9876 > logs/console.log &
7.4、访问rocketmq可视化系统
http://192.168.31.200:8080