RocketMQ安装配置与维护手册

准备工作

安装包准备

JAVA安装包

下载地址见https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html

RocketMQ安装包

下载地址见https://github.com/apache/rocketmq/releases

安装配置

组件 版本
JDK 8u241
RocketMQ 4.6.1

单机版

机器IP 操作系统版本 安装目录
192.168.229.121 CentOS-7.5 /middleware

注意事项

  namesrv默认配置JVM内存为4G,broker默认配置JVM内存为8G,如需修改,可以修改rocketmq家目录下bin下的runserver.sh和runbroker.sh,如下图所示:

runserver.sh


image.png

runbroker.sh


image.png

安装步骤

  1. 安装配置JAVA环境
  • 解压jdk压缩包到指定目录
[root@CentOS7 ~]# tar -xf jdk-8u241-linux-x64.tar.gz
[root@CentOS7 ~]# mv jdk1.8.0_241 /usr/local/
  • 设置环境变量
[root@CentOS7 ~]# echo "export $JAVA_HOME=/usr/local/jdk1.8.0_241" >> /etc/profile
[root@CentOS7 ~]# echo "export $PATH=$JAVA_HOME/bin:$PATH >> /etc/profile
  1. 安装RocketMQ

拷贝rocketmq安装包到指定目录,此处为/middleware/rocketmq-4.6.1

  • 启动namesrv
[root@CentOS7 ~]# nohup /middleware/rocketmq-4.6.1/bin/mqnamesrv &
  • 安装broker

修改broker配置文件

[root@CentOS7 rocketmq-4.6.1]# cat /middleware/rocketmq-4.6.1/conf/broker.conf
brokerClusterName=DefaultCluster
brokerName=broker-a
brokerIP1=192.168.229.121
brokerId=0
deleteWhen=04
fileReservedTime=48
storePathRootDir=/middleware/rocketmq/store
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
namesrvAddr=192.168.229.121:9876

启动broker

[root@CentOS7 ~]# nohup /middleware/rocketmq-4.6.1/bin/mqbroker -c /middleware/rocketmq-4.6.1/conf/broker.conf &
  • 验证
[root@CentOS7 ~]# /middleware/rocketmq-4.6.1/bin/mqadmin cluster -n 192.168.229.121:9876
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
#Cluster Name     #Broker Name            #BID  #Addr                  #Version                #InTPS(LOAD)       #OutTPS(LOAD) #PCWait(ms) #Hour #SPACE
DefaultCluster    broker-a                0     192.168.229.121:10911  V4_6_1                   0.00(0,0ms)         0.00(0,0ms)          0 447650.86 1.0000

集群版

机器IP 操作系统版本 安装目录 角色
192.168.229.121 CentOS-7.5 /middleware
192.168.229.128 CentOS-7.5 /middleware

注意事项

  1. namesrv默认配置JVM内存为4G,broker默认配置JVM内存为8G,修改方式同单机版修改方式
  2. 集群版默认提供了3个配置文件,分别是2主2从同步复制,2主2从异步复制和双主模式,实际部署可以根据自己的需要修改对应的配置文件

安装步骤

分别在2台机器上执行下列操作

  1. 安装JAVA环境
  • 解压jdk压缩包到指定目录
[root@CentOS7 ~]# tar -xf jdk-8u241-linux-x64.tar.gz
[root@CentOS7 ~]# mv jdk1.8.0_241 /usr/local/
  • 设置环境变量
[root@CentOS7 ~]# echo "export $JAVA_HOME=/usr/local/jdk1.8.0_241" >> /etc/profile
[root@CentOS7 ~]# echo "export $PATH=$JAVA_HOME/bin:$PATH >> /etc/profile
  1. 安装RocketMQ

拷贝rocketmq安装包到指定目录,此处为/middleware/rocketmq-4.6.1

  • 启动namesrv
[root@CentOS7 ~]# nohup /middleware/rocketmq-4.6.1/bin/mqnamesrv &
  • 安装broker

修改broker配置文件
192.168.229.128-主broker的配置文件为

[root@cent7-2 ~]# cat /middleware/rocketmq-4.6.1/conf/broker-10-m.conf
brokerClusterName=cluster1
brokerIP1=192.168.229.128
brokerIP2=192.168.229.128
brokerName=broker-10
brokerId=0
deleteWhen=04
fileReservedTime=48
storePathRootDir=/middleware/rocketmq/store
brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
namesrvAddr=192.168.229.128:9876;192.168.229.121:9876

192.168.229.121-从broker的配置文件为

[root@CentOS7 conf]# cat /middleware/rocketmq-4.6.1/conf/broker-10-s.conf
brokerClusterName=cluster1
brokerIP1=192.168.229.121
brokerIP2=192.168.229.121
brokerName=broker-10
brokerId=1
deleteWhen=04
fileReservedTime=48
storePathRootDir=/middleware/rocketmq/store
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
namesrvAddr=192.168.229.128:9876;192.168.229.121:9876

192.168.229.128启动broker

[root@CentOS7 ~]# nohup /middleware/rocketmq-4.6.1/bin/mqbroker -c /middleware/rocketmq-4.6.1/conf/broker-10-m.conf &

192.168.229.121启动broker

[root@CentOS7 ~]# nohup /middleware/rocketmq-4.6.1/bin/mqbroker -c /middleware/rocketmq-4.6.1/conf/broker-10-s.conf &

broker配置文件说明

参数值 说明
brokerClusterName 集群名称,一个集群内所有的broker机器该值配置相同
brokerIP1 broker服务监听IP
brokerIP2 存在broker主从时,在broker主节点上配置了brokerIP2的话,broker从节点会连接主节点配置的brokerIP2来同步,如果存在多块网卡,最好配置brokerIP1和brokerIP2,指定集群需要通信的IP地址
brokerName broker名称,主broker和从broker通过使用相同的Broker名称来表明相互关系
brokerId 一个主broker可以有多个从broker ,0表示是主,大于0表示从broker
deleteWhen 删除过期消息文件的时间
fileReservedTime 保存消息的时长,单位是小时,自动删除超时的消息,与deleteWhen共用
storePathRootDir 消息文件的保存目录
brokerRole SYNC_MASTER: 同步复制,数据会同步到slave一份,master和slave均写入成功后才反馈给生产者写成功状态;ASYNC_MASTER: 异步复制,数据会异步的形式发送到slave一份,master写入成功后即反馈给生产者写成功状态;SLAVE: 从broker
flushDiskType SYNC_FLUSH: 同步刷盘,消息刷到磁盘成功后才返回写成功状态;ASYNC_FLUSH: 异步刷盘,消息刷到pagecache后即返回写成功状态
namesrvAddr namesrv的地址
  1. 安装验证
[root@cent7-2 ~]# /middleware/rocketmq-4.6.1/bin/mqadmin clusterlist -n "192.168.229.121:9876;192.168.229.128:9876"
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
#Cluster Name     #Broker Name            #BID  #Addr                  #Version                #InTPS(LOAD)       #OutTPS(LOAD) #PCWait(ms) #Hour #SPACE
cluster1          broker-10               0     192.168.229.128:10911  V4_6_1                   0.00(0,0ms)         0.00(0,0ms)          0 5.93 1.0000
cluster1          broker-10               1     192.168.229.121:10911  V4_6_1                   0.00(0,0ms)         0.00(0,0ms)          0 5.93 1.0000

RocketMQ优化实践

内核参数优化

命令 说明
echo 'ulimit -n 655350' >> /etc/profile 设置当前用户可以打开的最大文件描述符数量,重启失效
echo '* hard nofile 655350' >> /etc/security/limits.conf 设置任一用户可以打开的最大文件描述符数量
echo '* hard memlock unlimited' >> /etc/security/limits.conf 最大锁定内存地址空间硬限制,避免内存被操作系统swap掉
echo '* soft memlock unlimited' >> /etc/security/limits.conf 最大锁定内存地址空间软限制,避免内存被操作系统swap掉
echo 'deadline' > /sys/block/${DISK}/queue/scheduler 设置磁盘的调度策略为deadline
sysctl -w vm.overcommit_memory=1 控制是否允许内存overcommit。设为1,则是允许。当应用申请内存时,系统都会认为存在足够的内存,准许申请
sysctl -w vm.drop_caches=1 设置为1会释放page cache。这个操作是一次性的,在执行该命令时释放page cache,没有持续性作用
sysctl -w vm.zone_reclaim_mode=0 该配置用于控制zone内存使用完之后的处理策略,设为0则禁止zone reclaim。对于大量使用缓存的应用来说,一般都需要禁止掉
sysctl -w vm.max_map_count=655360 RocketMQ使用MMAP映射文件到内存,因此需要设置映射文件的数量,避免MMAP操作失败
sysctl -w vm.dirty_background_ratio=50 设置内存中的脏数据占比,当超过该值时,内核后台线程将数据脏数据刷入磁盘
sysctl -w vm.dirty_ratio=50 设置内存的脏数据占比,当超过该值时,应用进程会主动将数据刷入磁盘
sysctl -w vm.dirty_writeback_centisecs=360000 内核线程会定期启动将内存中的旧数据刷入磁盘,通过此参数可以控制启动间隔
sysctl -w vm.page-cluster=3 设置一次读操作会加载几个数据页。
sysctl -w vm.swappiness=1 vm.swappiness用于控制使用系统swap空间比例,一般设置为0是禁止使用swap,设为1估计是某些系统不支持设置为0.RocketMQ的数据存储基于MMAP,大量使用内存,因此需要禁止swap,否则性能会急剧下降

broker配置优化

如果场景对消息可靠性要求比较严格,可以采取同步复制,同步刷盘模式,但是该种模式对性能影响比较大

主broker:
brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
从broker:
brokerRole=SLAVE
flushDiskType=SYNC_FLUSH

如果追求性能,可以采取异步复制异步刷盘模式,该种模式性能最高,但是消息可靠性最差

主broker:
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
从broker:
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH

在生产环境建议采用同步复制,异步刷盘模式,兼顾性能和消息可靠性

主broker:
brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
从broker:
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH

RocketMQ日常维护操作

查看集群状态

sh mqadmin clusterlist -n 'namesrv地址:端口'

[root@CentOS7 bin]# sh mqadmin clusterlist -n "192.168.229.121:9876;192.168.229.128:9876"
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
#Cluster Name     #Broker Name            #BID  #Addr                  #Version                #InTPS(LOAD)       #OutTPS(LOAD) #PCWait(ms) #Hour #SPACE
cluster1          broker-10               0     192.168.229.128:10911  V4_6_1                   0.00(0,0ms)         0.00(0,0ms)          0 8.88 1.0000
cluster1          broker-10               1     192.168.229.121:10911  V4_6_1                   0.00(0,0ms)         0.00(0,0ms)          0 8.88 1.0000

输出说明:
BID: broker的ID,0为主,其他的为从
Addr: broker的地址
Version: RocketMQ版本号
InTPS(LOAD): 生产者每秒钟发送的消息数量(发送线程池队列头部元素等待时间)
OutTPS(LOAD): 消费者每秒钟消费的消息数量(拉取线程池队列头部元素等待时间)
PCWait(ms): 缓存锁定时间
Hour: Broker启动时间
SPACE: commitLog磁盘占比

查看消费进度

sh mqadmin consumerProgress -g 'cid或者gid' -n 'namesrv地址:端口'

[root@CentOS7 bin]# sh mqadmin consumerprogress -g CID_PRODRES_CHANGE_OLD_W_E_CA
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
JM.Log:INFO Init JM logger with Slf4jLoggerFactory success, sun.misc.Launcher$ExtClassLoader@4b1210ee
JM.Log:INFO Log root path: /home/admin/logs/
JM.Log:INFO Set diamond-client log path: /home/admin/logs/diamond-client
#Topic                            #Broker Name                      #QID  #Broker Offset        #Consumer Offset      #Diff                 #LastTime
%RETRY%CID_PRODRES_CHANGE_OLD_W_  broker11                          0     616327                616326                1                     2021-01-25 18:35:41
%RETRY%CID_PRODRES_CHANGE_OLD_W_  broker12                          0     275051                275051                0                     2021-01-25 18:35:28
PRODRES_CHANGE_OLD_W_E_CARD       broker11                          0     3379335               3379335               0                     2021-01-25 18:35:37
PRODRES_CHANGE_OLD_W_E_CARD       broker11                          1     3379825               3379825               0                     2021-01-25 18:35:13
PRODRES_CHANGE_OLD_W_E_CARD       broker11                          2     3381098               3381098               0                     2021-01-25 18:34:57
PRODRES_CHANGE_OLD_W_E_CARD       broker11                          3     3381333               3381333               0                     2021-01-25 18:35:25
PRODRES_CHANGE_OLD_W_E_CARD       broker11                          4     3378243               3378243               0                     2021-01-25 18:34:50
PRODRES_CHANGE_OLD_W_E_CARD       broker11                          5     3380287               3380287               0                     2021-01-25 18:35:30
PRODRES_CHANGE_OLD_W_E_CARD       broker11                          6     3379392               3379392               0                     2021-01-25 18:35:17
PRODRES_CHANGE_OLD_W_E_CARD       broker11                          7     3376878               3376878               0                     2021-01-25 18:32:45
PRODRES_CHANGE_OLD_W_E_CARD       broker12                          0     3339015               3339015               0                     2021-01-25 18:35:32
PRODRES_CHANGE_OLD_W_E_CARD       broker12                          1     3340017               3340017               0                     2021-01-25 18:34:11
PRODRES_CHANGE_OLD_W_E_CARD       broker12                          2     3339889               3339888               1                     2021-01-25 18:34:28
PRODRES_CHANGE_OLD_W_E_CARD       broker12                          3     3336453               3336453               0                     2021-01-25 18:34:32
PRODRES_CHANGE_OLD_W_E_CARD       broker12                          4     3340674               3340674               0                     2021-01-25 18:35:25
PRODRES_CHANGE_OLD_W_E_CARD       broker12                          5     3337425               3337425               0                     2021-01-25 18:34:42
PRODRES_CHANGE_OLD_W_E_CARD       broker12                          6     3340160               3340160               0                     2021-01-25 18:35:05
PRODRES_CHANGE_OLD_W_E_CARD       broker12                          7     3338999               3338999               0                     2021-01-25 18:34:30

Consume TPS: 0.6499999999999999
Diff Total: 2

输出说明:
Topic: topic名称
Qid: 队列id
Broker offset: broker消息偏移量
Cousumer offset: 消费者消费消息偏移量(Consumer offset-Broker offset为堆积量)
Diff: 每个队列的堆积量
Consume TPS: 该组消费者每秒钟消费的消息数量
Diff Total: 该组消费者总的堆积量

查看broker配置参数

sh mqadmin getBrokerConfig -b 'broker地址:端口'

[root@CentOS7 bin]# sh mqadmin getBrokerConfig -b 192.168.229.121:10911
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
============192.168.229.121:10911============
serverSelectorThreads                             =  3
brokerRole                                        =  SLAVE
serverSocketRcvBufSize                            =  131072
osPageCacheBusyTimeOutMills                       =  1000
shortPollingTimeMills                             =  1000
clientSocketRcvBufSize                            =  131072
clusterTopicEnable                                =  true
brokerTopicEnable                                 =  true
autoCreateTopicEnable                             =  true
maxErrorRateOfBloomFilter                         =  20
......

修改broker配置参数

禁止broker写入

sh mqadmin updateBrokerConfig -b 'broker地址:端口' -k brokerPermission -v 4

[root@CentOS7 bin]# sh mqadmin updateBrokerConfig -b 192.168.229.121:10911 -k brokerPermission -v 4
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
update broker config success, 192.168.229.121:10911

允许broker写入

sh mqadmin updateBrokerConfig -b 'broker地址:端口' -k brokerPermission -v 6

[root@CentOS7 bin]# sh mqadmin updateBrokerConfig -b 192.168.229.121:10911 -k brokerPermission -v 6
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
update broker config success, 192.168.229.121:10911

修改发送队列等待时间

sh mqadmin updateBrokerConfig -c '集群名称' -k waitTimeMillsInSendQueue -v '值'

[root@CentOS7 bin]# sh mqadmin updateBrokerConfig -c cluster1 -k waitTimeMillsInSendQueue -v 2700 -n "192.168.229.121:9876;192.168.229.128:9876"
RocketMQLog:WARN No appenders could be found for logger (io.netty.util.internal.PlatformDependent0).
RocketMQLog:WARN Please initialize the logger system properly.
update broker config success, 192.168.229.128:10911

查询消费者的通道分配情况

[root@CentOS7 bin]# sh mqadmin consumerstatus -g CID_PRODACT_KFK_CHARGE
-bash-4.1$ sh mqadmin consumerstatus -g CID_PRODRES_CHANGE_OLD_W_E_CA
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
JM.Log:INFO Init JM logger with Slf4jLoggerFactory success, sun.misc.Launcher$ExtClassLoader@4b1210ee
JM.Log:INFO Log root path: /home/admin/logs/
JM.Log:INFO Set diamond-client log path: /home/admin/logs/diamond-client
001  10.191.6.123@29201#-72148370#-2113662258#16854876802355494 V3_6_2_SNAPSHOT      1611649872403/10.191.6.123@29201#-72148370#-2113662258#16854876802355494
002  10.191.32.43@15595#-72148370#-2113662258#16856843059515866 V3_6_2_SNAPSHOT      1611649872403/10.191.32.43@15595#-72148370#-2113662258#16856843059515866
003  10.191.32.42@12626#-72148370#-2113662258#16853156057574522 V3_6_2_SNAPSHOT      1611649872403/10.191.32.42@12626#-72148370#-2113662258#16853156057574522
004  10.191.32.51@11868#-72148370#-2113662258#4147806882227359 V3_6_2_SNAPSHOT      1611649872403/10.191.32.51@11868#-72148370#-2113662258#4147806882227359
005  10.191.6.127@15336#-72148370#-2113662258#16851112540028361 V3_6_2_SNAPSHOT      1611649872403/10.191.6.127@15336#-72148370#-2113662258#16851112540028361
006  10.191.6.126@15109#-72148370#-2113662258#16851128433606829 V3_6_2_SNAPSHOT      1611649872403/10.191.6.126@15109#-72148370#-2113662258#16851128433606829
007  10.191.6.129@23882#-72148370#-2113662258#16600532628304138 V3_6_2_SNAPSHOT      1611649872403/10.191.6.129@23882#-72148370#-2113662258#16600532628304138
008  10.191.6.125@3824#-72148370#-2113662258#16854816388973399 V3_6_2_SNAPSHOT      1611649872403/10.191.6.125@3824#-72148370#-2113662258#16854816388973399
009  10.191.6.122@6276#-72148370#-2113662258#16854877719952652 V3_6_2_SNAPSHOT      1611649872403/10.191.6.122@6276#-72148370#-2113662258#16854877719952652
010  10.191.32.44@11632#-72148370#-2113662258#16856946745907441 V3_6_2_SNAPSHOT      1611649872403/10.191.32.44@11632#-72148370#-2113662258#16856946745907441
011  10.191.6.124@16279#-72148370#-2113662258#16854800476860500 V3_6_2_SNAPSHOT      1611649872403/10.191.6.124@16279#-72148370#-2113662258#16854800476860500
012  10.191.32.53@24030#-72148370#-2113662258#16856847451283931 V3_6_2_SNAPSHOT      1611649872403/10.191.32.53@24030#-72148370#-2113662258#16856847451283931
013  10.191.32.50@16649#-72148370#-2113662258#16853127473470245 V3_6_2_SNAPSHOT      1611649872403/10.191.32.50@16649#-72148370#-2113662258#16853127473470245
014  10.191.32.52@14362#-72148370#-2113662258#16856838700264761 V3_6_2_SNAPSHOT      1611649872403/10.191.32.52@14362#-72148370#-2113662258#16856838700264761
015  10.191.6.128@15048#-72148370#-2113662258#16525614251821957 V3_6_2_SNAPSHOT      1611649872403/10.191.6.128@15048#-72148370#-2113662258#16525614251821957


Same subscription in the same group of consumer

Rebalance OK

[root@CentOS7 bin]# cd 1611649872403/
[root@CentOS7 bin]# cat * | grep ProcessQueueInfo * |sed  '1d'|awk '{print $1, $2, $3}'|sort -k2,3|grep -v "Broker Name"|awk -F[\#\:] '{print $1,$NF }'
10.191.32.42@12626 PRODRES_CHANGE_OLD_W_E_CARD broker11 0
10.191.32.42@12626 %RETRY%CID_PRODRES_CHANGE_OLD_W_E_CA broker11 0
10.191.32.42@12626 PRODRES_CHANGE_OLD_W_E_CARD broker11 1
10.191.32.43@15595 PRODRES_CHANGE_OLD_W_E_CARD broker11 2
10.191.32.44@11632 PRODRES_CHANGE_OLD_W_E_CARD broker11 3
10.191.32.50@16649 PRODRES_CHANGE_OLD_W_E_CARD broker11 4
10.191.32.51@11868 PRODRES_CHANGE_OLD_W_E_CARD broker11 5
10.191.32.52@14362 PRODRES_CHANGE_OLD_W_E_CARD broker11 6
10.191.32.53@24030 PRODRES_CHANGE_OLD_W_E_CARD broker11 7
10.191.32.43@15595 %RETRY%CID_PRODRES_CHANGE_OLD_W_E_CA broker12 0
10.191.6.122@6276 PRODRES_CHANGE_OLD_W_E_CARD broker12 0
10.191.6.123@29201 PRODRES_CHANGE_OLD_W_E_CARD broker12 1
10.191.6.124@16279 PRODRES_CHANGE_OLD_W_E_CARD broker12 2
10.191.6.125@3824 PRODRES_CHANGE_OLD_W_E_CARD broker12 3
10.191.6.126@15109 PRODRES_CHANGE_OLD_W_E_CARD broker12 4
10.191.6.127@15336 PRODRES_CHANGE_OLD_W_E_CARD broker12 5
10.191.6.128@15048 PRODRES_CHANGE_OLD_W_E_CARD broker12 6
10.191.6.129@23882 PRODRES_CHANGE_OLD_W_E_CARD broker12 7

重置消息位点

sh mqadmin resetOffsetByTime -g 'CID名称' -s now -t 'TOPIC名称'

如果已经进入retry队列,则需要通过以下命令,在broker或nameserv所在主机使用如下命令重置消费位点

sh mqadmin resetOffsetByTime -g 'CID名称' -s now -f true -t 'TOPIC名称'

领取限量云产品优惠券,让您在购买阿里云产品时能够享受最新的优惠和折扣,点击跳转到阿里云最新优惠活动页面

你可能感兴趣的:(RocketMQ安装配置与维护手册)