RocketMQ原理解析-Consumer

阅读更多
1. 介绍
  Consumer默认使用DefaultMQPushConsumerImpl长轮询拉消息的方式来消费,可以保证实时性同Push一样。还可以参考example项目,使用DefaultMQPullConsumerImpl,由业务控制如果拉消息,更新消费进度等方式。

2. 集群模式VS广播模式
  集群模式:
  • 消息的消费进度,即consumerOffset.json保存在broker上。
  • 所有consumer平均消费topic的消息。
  • 消息消费失败后,consumer会发回broker,broker根据消费失败次数设置不同的delayLevel进行重发。
  • 相同topic不同的consumerGroup组成伪广播模式,可达到所有consumer都会收到消息。

  广播模式:
  • 消息的消费进度保存在consumer的机器上。
  • 所有的consumer都会收到topic下的消息。
  • 消息消费失败后直接丢弃,不会发回broker进行重投递。


3. ConsumeFromWhere
  consumer可以设置消费的起始点,MQ提供了三种方式:
 
  • CONSUME_FROM_LAST_OFFSET
  • CONSUME_FROM_FIRST_OFFSET
  • CONSUME_FROM_TIMESTAMP

  我们从源码的角度,理解这三种不同消费点的含义。DefaultMQPushConsumerImpl内部有一个类RebalancePushImpl,首先计算客户端需要拉取队列,然后到broker获取消费进度,获取消费offset代码在computePullFromWhere方法:
public long computePullFromWhere(MessageQueue mq) {
        long result = -1;
        final ConsumeFromWhere consumeFromWhere = this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeFromWhere();
        final OffsetStore offsetStore = this.defaultMQPushConsumerImpl.getOffsetStore();
        switch (consumeFromWhere) {
            case CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST:
            case CONSUME_FROM_MIN_OFFSET:
            case CONSUME_FROM_MAX_OFFSET:
            case CONSUME_FROM_LAST_OFFSET: {
                long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
                if (lastOffset >= 0) {
                    result = lastOffset;
                }
                // First start,no offset
                else if (-1 == lastOffset) {
                    if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                        result = 0L;
                    } else {
                        try {
                            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
                        } catch (MQClientException e) {
                            result = -1;
                        }
                    }
                } else {
                    result = -1;
                }
                break;
            }
            case CONSUME_FROM_FIRST_OFFSET: {
                long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
                if (lastOffset >= 0) {
                    result = lastOffset;
                } else if (-1 == lastOffset) {
                    result = 0L;
                } else {
                    result = -1;
                }
                break;
            }
            case CONSUME_FROM_TIMESTAMP: {
                long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
                if (lastOffset >= 0) {
                    result = lastOffset;
                } else if (-1 == lastOffset) {
                    if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                        try {
                            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
                        } catch (MQClientException e) {
                            result = -1;
                        }
                    } else {
                        try {
                            long timestamp = UtilAll.parseDate(this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeTimestamp(),
                                    UtilAll.yyyyMMddHHmmss).getTime();
                            result = this.mQClientFactory.getMQAdminImpl().searchOffset(mq, timestamp);
                        } catch (MQClientException e) {
                            result = -1;
                        }
                    }
                } else {
                    result = -1;
                }
                break;
            }

            default:
                break;
        }

        return result;
    }

  首先看CONSUME_FROM_LAST_OFFSET的逻辑,lastOffset >= 0,意味着broker端有消费进度了,说明之前已经启动且消费了一些消息,那么就从返回的offset开始消费。当-1 == lastOffset,如果为重试队列,从头消费,为普通队列则从最大offset处消费。
  CONSUME_FROM_TIMESTAMP,如果为第一次启动,即-1 == lastOffset时,为普通队列的话,从设置的时间点开始消费,如果未设置时间点,默认从半小时前开始消费。
private String consumeTimestamp = UtilAll.timeMillisToHumanString3(System.currentTimeMillis() - (1000 * 60 * 30));


4. 拉消息
  Consumer的启动源码如下:
public void start() throws MQClientException {
        switch (this.serviceState) {
            case CREATE_JUST:
                log.info("the consumer [{}] start beginning. messageModel={}, isUnitMode={}", this.defaultMQPushConsumer.getConsumerGroup(),
                        this.defaultMQPushConsumer.getMessageModel(), this.defaultMQPushConsumer.isUnitMode());
                this.serviceState = ServiceState.START_FAILED;

                this.checkConfig();

                this.copySubscription();

                if (this.defaultMQPushConsumer.getMessageModel() == MessageModel.CLUSTERING) {
                    this.defaultMQPushConsumer.changeInstanceNameToPID();
                }

                this.mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);

                this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
                this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
                this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
                this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);

                this.pullAPIWrapper = new PullAPIWrapper(//
                        mQClientFactory, //
                        this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
                this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);

                if (this.defaultMQPushConsumer.getOffsetStore() != null) {
                    this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
                } else {
                    switch (this.defaultMQPushConsumer.getMessageModel()) {
                        case BROADCASTING:
                            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                            break;
                        case CLUSTERING:
                            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                            break;
                        default:
                            break;
                    }
                }
                this.offsetStore.load();

                if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
                    this.consumeOrderly = true;
                    this.consumeMessageService =
                            new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
                } else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
                    this.consumeOrderly = false;
                    this.consumeMessageService =
                            new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
                }

               this.consumeMessageService.start();

                boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
                if (!registerOK) {
                    this.serviceState = ServiceState.CREATE_JUST;
                    this.consumeMessageService.shutdown();
                    throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup()
                            + "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
                            null);
                }

               mQClientFactory.start();
                log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
                this.serviceState = ServiceState.RUNNING;
                break;
            case RUNNING:
            case START_FAILED:
            case SHUTDOWN_ALREADY:
                throw new MQClientException("The PushConsumer service state not OK, maybe started once, "//
                        + this.serviceState//
                        + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
                        null);
            default:
                break;
        }

        this.updateTopicSubscribeInfoWhenSubscriptionChanged();

        this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();

        this.mQClientFactory.rebalanceImmediately();
    }

this.offsetStore.load();

  首先就是加载消息消费进度,BROADCASTING模式实例化LocalFileOffsetStore,CLUSTERING实例化RemoteBrokerOffsetStore,加载内存,以后所有的读写都是直接操作的内存数据,由定时任务每隔5s进行落盘。
this.consumeMessageService.start();

  为消费消息服务,默认实现为ConsumeMessageConcurrentlyService,即并发消费。
mQClientFactory.start();

  mQClientFactory的启动会建立和broker通道,定时任务,拉消息服务,负载均衡服务。由于拉消息的请求是由负载均衡发起,所以先说负载均衡服务。
  负载均衡服务由RebalanceService线程每隔20s做一次,跟踪代码最终会调用到RebalanceImpl#rebalanceByTopic方法:
case BROADCASTING: {
                Set mqSet = this.topicSubscribeInfoTable.get(topic);
                if (mqSet != null) {
                    boolean changed = this.updateProcessQueueTableInRebalance(topic, mqSet, isOrder);
                    if (changed) {
                        this.messageQueueChanged(topic, mqSet, mqSet);
                        log.info("messageQueueChanged {} {} {} {}", //
                                consumerGroup, //
                                topic, //
                                mqSet, //
                                mqSet);
                    }
                } else {
                    log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
                }
                break;
            }

  广播模式,由于所有consumer都需要收到消息,所以不存在负载均衡策略。
List cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);

  集群模式下,首先通过topic和consumerGroup获取consumer列表,然后分配拉取消息队列,默认为平均分配策略AllocateMessageQueueAveragely。
RocketMQ原理解析-Consumer_第1张图片
  直接mock数据,可以看到AllocateMessageQueueAveragely的allocate方法,就是以上结论。
  得到当前consumer需要拉取的消息队列后,到RebalanceImpl#updateProcessQueueTableInRebalance进行拉取数据请求PullRequest构造,到DefaultMQPushConsumerImpl#pullMessage进行拉取消息前的逻辑整合,最终通过mQClientFactory内部的mQClientAPIImpl通道到broker异步拉取数据。
  回到DefaultMQPushConsumerImpl#pullMessage方法,看一下内部的逻辑:
long size = processQueue.getMsgCount().get();
        if (size > this.defaultMQPushConsumer.getPullThresholdForQueue()) {
            this.executePullRequestLater(pullRequest, PullTimeDelayMillsWhenFlowControl);
            if ((flowControlTimes1++ % 1000) == 0) {
                log.warn(
                        "the consumer message buffer is full, so do flow control, minOffset={}, maxOffset={}, size={}, pullRequest={}, flowControlTimes={}",
                        processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), size, pullRequest, flowControlTimes1);
            }
            return;
        }

  由于拉取的消息,会放到本地队列ProcessQueue进行处理,当发现本地队列大小超过1000,就延迟50ms再拉取。
 if (!this.consumeOrderly) {
            if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) {
                this.executePullRequestLater(pullRequest, PullTimeDelayMillsWhenFlowControl);
                if ((flowControlTimes2++ % 1000) == 0) {
                    log.warn(
                            "the queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, pullRequest={}, flowControlTimes={}",
                            processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(),
                            pullRequest, flowControlTimes2);
                }
                return;
            }
        }

  由于消息以offset为key,放入本地队列ProcessQueue的TreeMap中,所以这里有一步span检查。当span值(即this.msgTreeMap.lastKey() - this.msgTreeMap.firstKey())大于2000时,延迟拉取。 由于业务关系,消息消费快慢无法保证,如果offset大的消息处理的快,本地队列就会积压offset小的消息,所以span的值可能会越来越大。
  由于是异步拉取消息,所以这里需要构造一个PullCallback对象。onSuccess方法内:
if (pullResult != null) {
                    pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                            subscriptionData);

  对拉取结果进行处理,如果有消息,反序列化,再进行一步tag的比较去重。前面章节,有提到过,broker的ConsumeQueue中保存tag的hashcode值,所以consumer的此步去重是保证消息的准确性。
boolean dispathToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(//
                                        pullResult.getMsgFoundList(), //
                                        processQueue, //
                                        pullRequest.getMessageQueue(), //
                                        dispathToConsume);

  然后将消息放入本地队列,并通过submitConsumeRequest方法构造ConsumeRequest,由于consumeBatchSize为1,提交任务到consumeExecutor线程池(20个线程),每线程每消息并发处理,ConsumeRequest会调用consumeMessageService触发MessageListener#consumeMessage,执行业务处理。
ConsumeMessageConcurrentlyService.this.processConsumeResult(status, context, this);

  在ConsumeRequest#run业务处理完后,执行processConsumeResult方法:
switch (this.defaultMQPushConsumer.getMessageModel()) {
            case BROADCASTING:
                for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
                    MessageExt msg = consumeRequest.getMsgs().get(i);
                    log.warn("BROADCASTING, the message consume failed, drop it, {}", msg.toString());
                }
                break;
            case CLUSTERING:
                List msgBackFailed = new ArrayList(consumeRequest.getMsgs().size());
                for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
                    MessageExt msg = consumeRequest.getMsgs().get(i);
                    boolean result = this.sendMessageBack(msg, context);
                    if (!result) {
                        msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
                        msgBackFailed.add(msg);
                    }
                }

                if (!msgBackFailed.isEmpty()) {
                    consumeRequest.getMsgs().removeAll(msgBackFailed);

                    this.submitConsumeRequestLater(msgBackFailed, consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue());
                }
                break;
            default:
                break;
        }

  当业务处理返回RECONSUME_LATER后,ackIndex为-1,返回CONSUME_SUCCESS,ackIndex为0,。如果consumer为广播模式,消息消费失败后,直接打印log。如果consumer为集群模式,消息会通过this.sendMessageBack(msg, context)发回broker,consumer会重新收到消息。如果发回broker失败,consumer会尝试5s后重新消费。
  发回broker的这些消费失败消息会根据消费失败的次数设置不同的delayLevel。SendMessageProcessor#consumerSendMsgBack:
if (msgExt.getReconsumeTimes() >= maxReconsumeTimes//
                || delayLevel < 0) {
            newTopic = MixAll.getDLQTopic(requestHeader.getGroup());
            queueIdInt = Math.abs(this.random.nextInt() % 99999999) % DLQ_NUMS_PER_GROUP;

            topicConfig = this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(newTopic, //
                    DLQ_NUMS_PER_GROUP, //
                    PermName.PERM_WRITE, 0
            );
            if (null == topicConfig) {
                response.setCode(ResponseCode.SYSTEM_ERROR);
                response.setRemark("topic[" + newTopic + "] not exist");
                return response;
            }
        }

        else {
            if (0 == delayLevel) {
                delayLevel = 3 + msgExt.getReconsumeTimes();
            }

            msgExt.setDelayTimeLevel(delayLevel);
        }

  第一次消费失败,msgExt.getReconsumeTimes()为0,由于原始messageDelayLevel为“1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h”,当delayLevel为3的情况下,就对应10s,也就是说,第一次消费失败的消息,会在10s后重新消费,依次类推。消息自第一次消费失败后,由原先的topic转入%RETRY%队列,如果消息消费失败次数大于maxReconsumeTimes(16次),消息会进入DLQ队列。
 
long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
          this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), offset, true);
        }

  然后删除本地处理队列ProcessQueue中数据,注意这个removeMessage方法内部,永远返回的是msgTreeMap.firstKey(),所以更新消费的offset永远为最小值,这也正是MQ官方文档中说,MQ的重复消息需要让业务端过滤或采用幂等操作。
 
5. 长轮询
  前面提到,consumer是长轮询拉消息,当consumer拉消息时,broker端如果没有新消息,broker会通过PullRequestHoldService服务hold住这个请求:
if (brokerAllowSuspend && hasSuspendFlag) {
                        long pollingTimeMills = suspendTimeoutMillisLong;
                        if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) {
                            pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
                        }

                        String topic = requestHeader.getTopic();
                        long offset = requestHeader.getQueueOffset();
                        int queueId = requestHeader.getQueueId();
                        PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills,
                                this.brokerController.getMessageStore().now(), offset, subscriptionData);
                        this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);
                        response = null;
                        break;
                    }

  Broker通过ReputMessageService异步构建ConsumeQueue并通过注册的MessageArrivingListener通知PullRequestHoldService#notifyMessageArriving达到有消息,立即推送给consumer。ReputMessageService#doReput:
if (BrokerRole.SLAVE != DefaultMessageStore.this.getMessageStoreConfig().getBrokerRole()
                                            && DefaultMessageStore.this.brokerConfig.isLongPollingEnable()) {
                                        DefaultMessageStore.this.messageArrivingListener.arriving(dispatchRequest.getTopic(),
                                                dispatchRequest.getQueueId(), dispatchRequest.getConsumeQueueOffset() + 1,
                                                dispatchRequest.getTagsCode());
                                    }

public void arriving(String topic, int queueId, long logicOffset, long tagsCode) {
        this.pullRequestHoldService.notifyMessageArriving(topic, queueId, logicOffset, tagsCode);
    }
  • RocketMQ原理解析-Consumer_第2张图片
  • 大小: 72.7 KB
  • 查看图片附件

你可能感兴趣的:(RocketMQ,MQ,消息中间件)