RocketMQ的消息过滤分为两种:表达式过滤和类过滤。表达式过滤针对消息的属性过滤,适合于简单的场景,类过滤可以实现复杂的逻辑。
表达式过滤分为tag过滤和SQL92过滤。SQL92在这里不展开描述,只介绍下tag过滤。
在客户端发送消息的时候可以指定消息的tag,并根据消息的tag生成哈希值,为tagcode,存储在CommitLog中。所以tag过滤分为两部分:拉取过滤和消费过滤,分别发生在broker端和消费端。
消费者订阅消费组时,将消息的过滤保存,以便消息拉取时过滤使用。详细代码见:org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#subscribe(java.lang.String, java.lang.String)
public void subscribe(String topic, String subExpression) throws MQClientException {
try {
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPushConsumer.getConsumerGroup(),
topic, subExpression);
this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
if (this.mQClientFactory != null) {
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
}
} catch (Exception e) {
throw new MQClientException("subscription exception", e);
}
}
来看下SubscriptionData的核心属性:
public class SubscriptionData implements Comparable<SubscriptionData> {
public final static String SUB_ALL = "*"; //订阅所有消息
private boolean classFilterMode = false; // 类过滤模式标记
private String topic;
private String subString; //表达式,可用竖线隔开
private Set<String> tagsSet = new HashSet<String>();// tag集合
private Set<Integer> codeSet = new HashSet<Integer>();// tagcode集合
private long subVersion = System.currentTimeMillis();
private String expressionType = ExpressionType.TAG;// 过滤类型
}
根据上面的存储,生成拉取请求中的过滤标记,见org.apache.rocketmq.client.impl.consumer.PullAPIWrapper#pullKernelImpl
中构造requestHeader部分,这里不再贴源码。
根据请求requestHeader中的过滤信息,构造MessageFilter,见 org.apache.rocketmq.broker.processor.PullMessageProcessor#processRequest(io.netty.channel.Channel, org.apache.rocketmq.remoting.protocol.RemotingCommand, boolean)
MessageFilter messageFilter;
if (this.brokerController.getBrokerConfig().isFilterSupportRetry()) {
messageFilter = new ExpressionForRetryMessageFilter(subscriptionData, consumerFilterData,
this.brokerController.getConsumerFilterManager());
} else {
messageFilter = new ExpressionMessageFilter(subscriptionData, consumerFilterData,
this.brokerController.getConsumerFilterManager());
}
直接看最终的过滤逻辑,这里以ExpressionMessageFilter为例:
public boolean isMatchedByConsumeQueue(Long tagsCode, ConsumeQueueExt.CqExtUnit cqExtUnit) {
if (null == subscriptionData) {
return true;
}
if (subscriptionData.isClassFilterMode()) { //类过滤这里不处理
return true;
}
// tag模式
if (ExpressionType.isTagType(subscriptionData.getExpressionType())) {
if (tagsCode == null) {// tag为空
return true;
}
if (subscriptionData.getSubString().equals(SubscriptionData.SUB_ALL)) {// 全订阅
return true;
}
return subscriptionData.getCodeSet().contains(tagsCode.intValue()); // 包含即命中
} else {
// no expression or no bloom
if (consumerFilterData == null || consumerFilterData.getExpression() == null
|| consumerFilterData.getCompiledExpression() == null || consumerFilterData.getBloomFilterData() == null) {
return true;
}
// message is before consumer
if (cqExtUnit == null || !consumerFilterData.isMsgInLive(cqExtUnit.getMsgStoreTime())) {
log.debug("Pull matched because not in live: {}, {}", consumerFilterData, cqExtUnit);
return true;
}
byte[] filterBitMap = cqExtUnit.getFilterBitMap();
BloomFilter bloomFilter = this.consumerFilterManager.getBloomFilter();
if (filterBitMap == null || !this.bloomDataValid
|| filterBitMap.length * Byte.SIZE != consumerFilterData.getBloomFilterData().getBitNum()) {
return true;
}
BitsArray bitsArray = null;
try {
bitsArray = BitsArray.create(filterBitMap);
boolean ret = bloomFilter.isHit(consumerFilterData.getBloomFilterData(), bitsArray);
log.debug("Pull {} by bit map:{}, {}, {}", ret, consumerFilterData, bitsArray, cqExtUnit);
return ret;
} catch (Throwable e) {
log.error("bloom filter error, sub=" + subscriptionData
+ ", filter=" + consumerFilterData + ", bitMap=" + bitsArray, e);
}
}
return true;
}
因为在broker端过滤的时候只比较了tagcode,并没有比较tag的值,在消息消费的时候还需要对tag值做校验,具体逻辑详见:org.apache.rocketmq.client.impl.consumer.PullAPIWrapper#processPullResult
public PullResult processPullResult(final MessageQueue mq, final PullResult pullResult,
final SubscriptionData subscriptionData) {
PullResultExt pullResultExt = (PullResultExt) pullResult;
this.updatePullFromWhichNode(mq, pullResultExt.getSuggestWhichBrokerId());
if (PullStatus.FOUND == pullResult.getPullStatus()) {
ByteBuffer byteBuffer = ByteBuffer.wrap(pullResultExt.getMessageBinary());
List<MessageExt> msgList = MessageDecoder.decodes(byteBuffer);
List<MessageExt> msgListFilterAgain = msgList;
if (!subscriptionData.getTagsSet().isEmpty() && !subscriptionData.isClassFilterMode()) {
msgListFilterAgain = new ArrayList<MessageExt>(msgList.size());
for (MessageExt msg : msgList) {
if (msg.getTags() != null) {
// 对比tag值
if (subscriptionData.getTagsSet().contains(msg.getTags())) {
msgListFilterAgain.add(msg);
}
}
}
}
...
return pullResult;
}
在broker端的CommitLog过滤时,使用的是tagcode过滤,原因是CommitLog中只保存了tagcode。为什么只保留tagcode呢?是为了将CommitLog设计为定长的,提高存储效率。所以需要在消息真正消费的时候,再用消息的真实tag过滤一遍
待补充