背景


架构部署

32C  128G 1.8T 4主4从架构部署


rocketmq延迟服务生产者生产数据，消费者无法正常消费到数据，查看日志有重大发现，所有的主节点均有这报错

#vim  /data2/logs/rocketmqlogs/storeerror.log
020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:56 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 509548460, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1
2020-03-18 16:46:57 WARN ScheduleMessageTimerThread - Offset not matched. Request offset: 665682360, firstOffset: 0, lastOffset: 6000000, mappedFileSize: 6000000, mappedFiles count: 1

发现消费的位点全部错乱，导致消费者无法常消费到数据


延迟消息存储时被替换为“SCHEDULE_TOPIC_XXXX”主题，broker会为每个等级的建立定时任务进行调度，将各个等级到时间的消息替换为原来的主题，从而消费者可以消费到该消息。以上日志加上消息查询结果已存储在“SCHEDULE_TOPIC_XXXX”，可以断定调度环节出问题了，由于offset远远超过最大offset而报错；没能将目标主题成功替换。


解决方法

先将broker节点的流量清除，2表示只写权限，4表示只读权限，6表示读写权限

#bin/mqadmin updateBrokerConfig -b brokerip:10911 -n nameserver地址:9876 -k brokerPermission -v 4


查看broker节点流量是否为0,需要等待broker完全没有流量了才可以进行关闭broker节点操作，以防止丢数据

#bin/mqadmin clusterList -n nameserver:9876


关闭broker节点进程

#bin/mqshutdown broker


将delayOffset.json 和delayOffset.json.bak文件拷走 此文件用于记录消费位点的，拷走后，重启应用进程会自动生产

#mv /data1/rocketmq/store/config/delayOffset.json* /data2/


将SCHEDULE_TOPIC_XXXX 文件拷走，用于调度任务，拷走后，重启应用进程会自动生产

#mv /data1/rocketmq/store/consumequeue/SCHEDULE_TOPIC_XXXX/ /data2/


将rocketmq的日志文件全部删除

#rm -rf /data2/logs/rocketmqlogs/*


然后将rocketmq进程启动

#cd /usr/local/rocketmq/bin/

#nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-2s-async/broker-a.properties &


把权限重新赋值

#bin/mqadmin updateBrokerConfig -b brokerip:10911 -n nameserver地址:9876 -k brokerPermission -v 6


等节点流量恢复重，按上面的操作，将集群所有主节点都操作一遍，就能正常消费数据了，且storeerror.log日志也没在报错了