系列
- RocketMq 消息Tag过滤
- RocketMq 广播模式
- RocketMQ 同步调用的新特性
- RocketMq 事务消息
- RocketMq 堆积查询
- RocketMq 消息查询
- RocketMq 消费位点上报
开篇
- 这篇文章主要解析下RocketMq的两张消息查询方式，分别是queryMsgById和queryMsgByKey。
- queryMsgById根据MsgId查询消息体。
- queryMsgByKey根据key查询消息体。
- queryMsgByUniqueKey根据uniqueKey进行查询。
queryMsgById
usage: mqadmin queryMsgById [-d ] [-g ] [-h] -i [-n ] [-s ] [-u ]
-d,--clientId The consumer's client id
-g,--consumerGroup consumer group name
-h,--help Print help
-i,--msgId Message Id
-n,--namesrvAddr Name server address list, eg: 192.168.0.1:9876;192.168.0.2:9876
-s,--sendMessage resend message
-u,--unitName unit name
- queryMsgById根据msgId去查询实际消息内容，命令形式如上所示。
mqadmin queryMsgById -n localhost -i C0A8010300002A9F0000000000D80335
查询结果
OffsetID: C0A8010300002A9F0000000000D80335
Topic: SCHEDULE_TOPIC_XXXX
Tags: [TagA]
Keys: [unix_1]
Queue ID: 2
Queue Offset: 11795
CommitLog Offset: 14156597
Reconsume Times: 0
Born Timestamp: 2020-05-12 22:54:45,771
Store Timestamp: 2020-05-12 22:54:45,784
Born Host: 192.168.1.3:55209
Store Host: 192.168.1.3:10911
System Flag: 0
Properties: {REAL_TOPIC=TopicTest, KEYS=unix_1, UNIQ_KEY=24098A2802202740D87C1B9F2622006B000018B4AAC23D9096070000, CLUSTER=DefaultCluster, WAIT=true, DELAY=3, TAGS=TagA, REAL_QID=3}
Message Body Path: /tmp/rocketmq/msgbodys/24098A2802202740D87C1B9F2622006B000018B4AAC23D9096070000
- msgId如C0A8010300002A9F0000000000D80335，msgId由ip:port:offset三者组成。
public class MessageDecoder {
public static String createMessageId(final ByteBuffer input, final ByteBuffer addr, final long offset) {
input.flip();
// ByteBuffer addr包含ip和端口
// 针对ipv4地址，ip为4字节，端口为4字节，offset为8字节，总计16个字节。
// 针对ipv6地址，ip地址为16个字节，端口为4字节，offset为8字节，总结28个字节。
int msgIDLength = addr.limit() == 8 ? 16 : 28;
input.limit(msgIDLength);
input.put(addr);
input.putLong(offset);
return UtilAll.bytes2string(input.array());
}
public static MessageId decodeMessageId(final String msgId) throws UnknownHostException {
SocketAddress address;
long offset;
// ipv4地址为32bit，等于4个字节，等于8个16进制码
// ipv6地址为128bit，等于16个字节，等于32个16进制码
int ipLength = msgId.length() == 32 ? 4 * 2 : 16 * 2;
// 取除ip地址
byte[] ip = UtilAll.string2bytes(msgId.substring(0, ipLength));
// 取出port端口
byte[] port = UtilAll.string2bytes(msgId.substring(ipLength, ipLength + 8));
ByteBuffer bb = ByteBuffer.wrap(port);
int portInt = bb.getInt(0);
address = new InetSocketAddress(InetAddress.getByAddress(ip), portInt);
// 取出offset偏移
byte[] data = UtilAll.string2bytes(msgId.substring(ipLength + 8, ipLength + 8 + 16));
bb = ByteBuffer.wrap(data);
offset = bb.getLong(0);
return new MessageId(address, offset);
}
}
public class MQAdminImpl {
public MessageExt viewMessage(
String msgId) throws RemotingException, MQBrokerException, InterruptedException, MQClientException {
MessageId messageId = null;
try {
messageId = MessageDecoder.decodeMessageId(msgId);
} catch (Exception e) {
throw new MQClientException(ResponseCode.NO_MESSAGE, "query message by id finished, but no message.");
}
return this.mQClientFactory.getMQClientAPIImpl().viewMessage(RemotingUtil.socketAddress2String(messageId.getAddress()),
messageId.getOffset(), timeoutMillis);
}
}
- 针对ipv4地址，ip为4字节，端口为4字节，offset为8字节，总计16个字节。
- 针对ipv6地址，ip地址为16个字节，端口为4字节，offset为8字节，总结28个字节。
- createMessageId的组成包括ip:port:offset三者。
- viewMessage#decodeMessageId会解析成ip:port和offset，通过ip:port查询指定的broker，根据offset查询指定的commitLog的内容。
public class QueryMessageProcessor implements NettyRequestProcessor {
public RemotingCommand viewMessageById(ChannelHandlerContext ctx, RemotingCommand request)
throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(null);
final ViewMessageRequestHeader requestHeader =
(ViewMessageRequestHeader) request.decodeCommandCustomHeader(ViewMessageRequestHeader.class);
response.setOpaque(request.getOpaque());
// 根据offset去commitLog查询实际存储数据
final SelectMappedBufferResult selectMappedBufferResult =
this.brokerController.getMessageStore().selectOneMessageByOffset(requestHeader.getOffset());
if (selectMappedBufferResult != null) {
response.setCode(ResponseCode.SUCCESS);
response.setRemark(null);
try {
FileRegion fileRegion =
new OneMessageTransfer(response.encodeHeader(selectMappedBufferResult.getSize()),
selectMappedBufferResult);
ctx.channel().writeAndFlush(fileRegion).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture future) throws Exception {
selectMappedBufferResult.release();
if (!future.isSuccess()) {
log.error("Transfer one message from page cache failed, ", future.cause());
}
}
});
} catch (Throwable e) {
log.error("", e);
selectMappedBufferResult.release();
}
return null;
} else {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark("can not find message by the offset, " + requestHeader.getOffset());
}
return response;
}
}
- viewMessageById在根据requestHeader.getOffset()的位移去commitLog获取消息体。
queryMsgByKey
usage: mqadmin queryMsgByKey [-h] -k [-n ] -t
-h,--help Print help
-k,--msgKey Message Key
-n,--namesrvAddr Name server address list, eg: 192.168.0.1:9876;192.168.0.2:9876
-t,--topic topic name
- queryMsgByKey根据topic 和 msgKey 去查询实际消息内容，命令形式如上所示。
mqadmin queryMsgByKey -k unix_1 -t TopicTest -n localhost:9876
#Message ID #QID #Offset
C0A80008000018B4AAC23D6A91EA0000 7 1256
C0A80008000018B4AAC23D6BDB330000 0 1405
24098A2802202740D87C1B9F2622006B000018B4AAC23D7948F00000 3 1516
24098A2802202740D87C1B9F2622006B000018B4AAC23D7D2BA50000 6 1626
24098A2802202740D87C1B9F2622006B000018B4AAC23D7F844F0000 6 1751
24098A2802202740D87C1B9F2622006B000018B4AAC23D8089980000 0 1905
24098A2802202740D87C1B9F2622006B000018B4AAC23D81031F0000 0 1906
24098A2802202740D87C1B9F2622006B000018B4AAC23D81BBB00000 2 1873
24098A2802202740D87C1B9F2622006B000018B4AAC23D82D0BB0000 2 1874
24098A2802202740D87C1B9F2622006B000018B4AAC23D83B57F0000 0 1907
24098A2802202740D87C1B9F2622006B000018B4AAC23D850EC70000 0 1908
24098A2802202740D87C1B9F2622006B000018B4AAC23D8C99B10000 1 1909
24098A2802202740D87C1B9F2622006B000018B4AAC23D9096070000 3 1891
24098A2802202740D87C1B9F2622006B000018B4AAC23D9553AF0000 5 1867
24098A2802202740D87C1B9F2622006B000018B4AAC23D9640380000 1 1910
24098A2802202740D87C1B9F2622006B000018B4AAC23D96F68B0000 1 1911
24098A2802202740D87C1B9F2622006B000018B4AAC23D9822BE0000 2 1875
public class DefaultMessageStore implements MessageStore {
@Override
public QueryMessageResult queryMessage(String topic, String key, int maxNum, long begin, long end) {
QueryMessageResult queryMessageResult = new QueryMessageResult();
long lastQueryMsgTime = end;
for (int i = 0; i < 3; i++) {
// 去indexService根据topic和key去查询
QueryOffsetResult queryOffsetResult = this.indexService.queryOffset(topic, key, maxNum, begin, lastQueryMsgTime);
if (queryOffsetResult.getPhyOffsets().isEmpty()) {
break;
}
Collections.sort(queryOffsetResult.getPhyOffsets());
queryMessageResult.setIndexLastUpdatePhyoffset(queryOffsetResult.getIndexLastUpdatePhyoffset());
queryMessageResult.setIndexLastUpdateTimestamp(queryOffsetResult.getIndexLastUpdateTimestamp());
for (int m = 0; m < queryOffsetResult.getPhyOffsets().size(); m++) {
long offset = queryOffsetResult.getPhyOffsets().get(m);
try {
boolean match = true;
MessageExt msg = this.lookMessageByOffset(offset);
if (0 == m) {
lastQueryMsgTime = msg.getStoreTimestamp();
}
if (match) {
SelectMappedBufferResult result = this.commitLog.getData(offset, false);
if (result != null) {
int size = result.getByteBuffer().getInt(0);
result.getByteBuffer().limit(size);
result.setSize(size);
queryMessageResult.addMessage(result);
}
} else {
log.warn("queryMessage hash duplicate, {} {}", topic, key);
}
} catch (Exception e) {
log.error("queryMessage exception", e);
}
}
if (queryMessageResult.getBufferTotalSize() > 0) {
break;
}
if (lastQueryMsgTime < begin) {
break;
}
}
return queryMessageResult;
}
}
- 根据topic和key去indexService查询索引信息。
- 根据查询结果的phyoffset去commitLog查询实际存储的消息。
queryMsgByUniqueKey
queryMsgByUniqueKey -t TopicTest -n 192.168.0.8:9876 -i C0A80008000018B4AAC23DBBCAD00000
Topic: TopicTest
Tags: [TagA]
Keys: [unix_1]
Queue ID: 2
Queue Offset: 1876
CommitLog Offset: 14159198
Reconsume Times: 0
Born Timestamp: 2020-05-12 23:41:57,329
Store Timestamp: 2020-05-12 23:41:57,344
Born Host: 192.168.0.8:61405
Store Host: 192.168.0.8:10911
System Flag: 0
Properties: {KEYS=unix_1, UNIQ_KEY=C0A80008000018B4AAC23DBBCAD00000, CLUSTER=DefaultCluster, WAIT=true, TAGS=TagA}
Message Body Path: /tmp/rocketmq/msgbodys/C0A80008000018B4AAC23DBBCAD00000
MessageTrack [consumerGroup=order_consumer, trackType=NOT_ONLINE, exceptionDesc=CODE:206 DESC:the consumer group[order_consumer] not online]MessageTrack [consumerGroup=consumer_group_test, trackType=NOT_ONLINE, exceptionDesc=CODE:206 DESC:the consumer group[consumer_group_test] not online]MessageTrack [consumerGroup=ResponseConsumerGroup, trackType=NOT_ONLINE, exceptionDesc=CODE:206 DESC:the consumer group[ResponseConsumerGroup] not online]MessageTrack [consumerGroup=please_rename_unique_group_name_4, trackType=UNKNOWN, exceptionDesc=org.apache.rocketmq.client.exception.MQClientException: CODE: 17 DESC: No topic route info in name server for the topic: %RETRY%please_rename_unique_group_name_4
See http://rocketmq.apache.org/docs/faq/ for further details., org.apache.rocketmq.client.impl.MQClientAPIImpl.getTopicRouteInfoFromNameServer(MQClientAPIImpl.java:1389)]
- producer发送消息后返回的msgId=C0A80008000018B4AAC23DC95F940000本质上是UniqueKey，而非queryMsgById中使用的Id
