RocketMQ消息消费六：消息过滤

概述

RocketMQ的消息过滤分为两种：表达式过滤和类过滤。表达式过滤针对消息的属性过滤，适合于简单的场景，类过滤可以实现复杂的逻辑。

表达式过滤

表达式过滤分为tag过滤和SQL92过滤。SQL92在这里不展开描述，只介绍下tag过滤。
在客户端发送消息的时候可以指定消息的tag，并根据消息的tag生成哈希值，为tagcode，存储在CommitLog中。所以tag过滤分为两部分：拉取过滤和消费过滤，分别发生在broker端和消费端。

拉取过滤

消费者过滤规则存储

消费者订阅消费组时，将消息的过滤保存，以便消息拉取时过滤使用。详细代码见：org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#subscribe(java.lang.String, java.lang.String)

    public void subscribe(String topic, String subExpression) throws MQClientException {
        try {
            SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPushConsumer.getConsumerGroup(),
                topic, subExpression);
            this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
            if (this.mQClientFactory != null) {
                this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
            }
        } catch (Exception e) {
            throw new MQClientException("subscription exception", e);
        }
    }

来看下SubscriptionData的核心属性：

public class SubscriptionData implements Comparable<SubscriptionData> {
    public final static String SUB_ALL = "*"; //订阅所有消息
    private boolean classFilterMode = false; // 类过滤模式标记
    private String topic;
    private String subString; //表达式，可用竖线隔开
    private Set<String> tagsSet = new HashSet<String>();// tag集合
    private Set<Integer> codeSet = new HashSet<Integer>();// tagcode集合
    private long subVersion = System.currentTimeMillis();
    private String expressionType = ExpressionType.TAG;// 过滤类型
}

消费者拉取消息时生成过滤标记

根据上面的存储，生成拉取请求中的过滤标记，见org.apache.rocketmq.client.impl.consumer.PullAPIWrapper#pullKernelImpl
中构造requestHeader部分，这里不再贴源码。

broker过滤tagcode

根据请求requestHeader中的过滤信息，构造MessageFilter，见 org.apache.rocketmq.broker.processor.PullMessageProcessor#processRequest(io.netty.channel.Channel, org.apache.rocketmq.remoting.protocol.RemotingCommand, boolean)

        MessageFilter messageFilter;
        if (this.brokerController.getBrokerConfig().isFilterSupportRetry()) {
            messageFilter = new ExpressionForRetryMessageFilter(subscriptionData, consumerFilterData,
                this.brokerController.getConsumerFilterManager());
        } else {
            messageFilter = new ExpressionMessageFilter(subscriptionData, consumerFilterData,
                this.brokerController.getConsumerFilterManager());
        }

直接看最终的过滤逻辑，这里以ExpressionMessageFilter为例：

public boolean isMatchedByConsumeQueue(Long tagsCode, ConsumeQueueExt.CqExtUnit cqExtUnit) {
        if (null == subscriptionData) {
            return true;
        }

        if (subscriptionData.isClassFilterMode()) { //类过滤这里不处理
            return true;
        }

        // tag模式
        if (ExpressionType.isTagType(subscriptionData.getExpressionType())) {

            if (tagsCode == null) {// tag为空
                return true;
            }

            if (subscriptionData.getSubString().equals(SubscriptionData.SUB_ALL)) {// 全订阅
                return true;
            }

            return subscriptionData.getCodeSet().contains(tagsCode.intValue()); // 包含即命中
        } else {
            // no expression or no bloom
            if (consumerFilterData == null || consumerFilterData.getExpression() == null
                || consumerFilterData.getCompiledExpression() == null || consumerFilterData.getBloomFilterData() == null) {
                return true;
            }

            // message is before consumer
            if (cqExtUnit == null || !consumerFilterData.isMsgInLive(cqExtUnit.getMsgStoreTime())) {
                log.debug("Pull matched because not in live: {}, {}", consumerFilterData, cqExtUnit);
                return true;
            }

            byte[] filterBitMap = cqExtUnit.getFilterBitMap();
            BloomFilter bloomFilter = this.consumerFilterManager.getBloomFilter();
            if (filterBitMap == null || !this.bloomDataValid
                || filterBitMap.length * Byte.SIZE != consumerFilterData.getBloomFilterData().getBitNum()) {
                return true;
            }

            BitsArray bitsArray = null;
            try {
                bitsArray = BitsArray.create(filterBitMap);
                boolean ret = bloomFilter.isHit(consumerFilterData.getBloomFilterData(), bitsArray);
                log.debug("Pull {} by bit map:{}, {}, {}", ret, consumerFilterData, bitsArray, cqExtUnit);
                return ret;
            } catch (Throwable e) {
                log.error("bloom filter error, sub=" + subscriptionData
                    + ", filter=" + consumerFilterData + ", bitMap=" + bitsArray, e);
            }
        }

        return true;
    }

消费过滤

因为在broker端过滤的时候只比较了tagcode，并没有比较tag的值，在消息消费的时候还需要对tag值做校验，具体逻辑详见：org.apache.rocketmq.client.impl.consumer.PullAPIWrapper#processPullResult

    public PullResult processPullResult(final MessageQueue mq, final PullResult pullResult,
        final SubscriptionData subscriptionData) {
        PullResultExt pullResultExt = (PullResultExt) pullResult;

        this.updatePullFromWhichNode(mq, pullResultExt.getSuggestWhichBrokerId());
        if (PullStatus.FOUND == pullResult.getPullStatus()) {
            ByteBuffer byteBuffer = ByteBuffer.wrap(pullResultExt.getMessageBinary());
            List<MessageExt> msgList = MessageDecoder.decodes(byteBuffer);

            List<MessageExt> msgListFilterAgain = msgList;
            if (!subscriptionData.getTagsSet().isEmpty() && !subscriptionData.isClassFilterMode()) {
                msgListFilterAgain = new ArrayList<MessageExt>(msgList.size());
                for (MessageExt msg : msgList) {
                    if (msg.getTags() != null) {
                    // 对比tag值
                        if (subscriptionData.getTagsSet().contains(msg.getTags())) {
                            msgListFilterAgain.add(msg);
                        }
                    }
                }
            }

			...

        return pullResult;
    }

小结

在broker端的CommitLog过滤时，使用的是tagcode过滤，原因是CommitLog中只保存了tagcode。为什么只保留tagcode呢？是为了将CommitLog设计为定长的，提高存储效率。所以需要在消息真正消费的时候，再用消息的真实tag过滤一遍

类过滤模式

待补充