RocketMQ源码（一）：NameServer的启动
RocketMQ源码（二）：broker的启动（一）
RocketMQ源码（三）：broker的启动（二）
RocketMQ源码（四）：producer的启动
RocketMQ源码（五）：producer发送消息
RocketMQ源码（六）：broker接收消息
RocketMQ源码（七）：consumer的启动
RocketMQ源码（九）：consumer消息拉取（二）
消息拉取整体流程
以上就是消息拉取从consumer到broker的整体流程，拆分成consumer端和broker端两部分来分析这个过程，这一片文章先看下consumer端所做的事情
上一章中留下过一个地方没有展开说明，RebalanceImpl的rebalanceByTopic方法中，在消费队列分配完成后，通过MessageQueueListener将消费的messageQueue变更事件通知出去。
这里来看下MessageQueueListener，MessageQueueListener有两个实现类其中一个即将被移除，因此来看下DefaultLitePullConsumerImpl的内部类MessageQueueListenerImpl。
在调用subscribe时，会注册一个MessageQueueListenerImpl用于监听consumer对应消费的MessageQueue变化，当RebalanceLitePullImpl在rebalanceByTopic方法中判断MessageQueue变化时就会调用MessageQueueListenerImpl。
class MessageQueueListenerImpl implements MessageQueueListener {
@Override
public void messageQueueChanged(String topic, Set mqAll, Set mqDivided) {
MessageModel messageModel = defaultLitePullConsumer.getMessageModel();
switch (messageModel) {
// 广播模式
case BROADCASTING:
updateAssignedMessageQueue(topic, mqAll);
updatePullTask(topic, mqAll);
break;
// 集群模式
case CLUSTERING:
// 更新订阅的 MessageQueue
updateAssignedMessageQueue(topic, mqDivided);
updatePullTask(topic, mqDivided);
break;
default:
break;
}
}
}
可以看到的是这里主要是两个部分
- 更新AssignedMessageQueue中，订阅的messageQueue信息
- 开始拉取消息任务
首先看下第一部分的代码
private void updateAssignedMessageQueue(String topic, Set assignedMessageQueue) {
this.assignedMessageQueue.updateAssignedMessageQueue(topic, assignedMessageQueue);
}
public void updateAssignedMessageQueue(String topic, Collection assigned) {
synchronized (this.assignedMessageQueueState) {
Iterator> it = this.assignedMessageQueueState.entrySet().iterator();
// 移除没有被分配到的processQueue
while (it.hasNext()) {
Map.Entry next = it.next();
if (next.getKey().getTopic().equals(topic)) {
if (!assigned.contains(next.getKey())) {
next.getValue().getProcessQueue().setDropped(true);
it.remove();
}
}
}
// 保存新的messageQueue
addAssignedMessageQueue(assigned);
}
}
private void addAssignedMessageQueue(Collection assigned) {
for (MessageQueue messageQueue : assigned) {
// 如果是新增的messageQueue
if (!this.assignedMessageQueueState.containsKey(messageQueue)) {
MessageQueueState messageQueueState;
// 新建或者使用以前的ProcessQueue，新建一个MessageQueueState然后保存
if (rebalanceImpl != null && rebalanceImpl.getProcessQueueTable().get(messageQueue) != null) {
messageQueueState = new MessageQueueState(messageQueue, rebalanceImpl.getProcessQueueTable().get(messageQueue));
} else {
ProcessQueue processQueue = new ProcessQueue();
messageQueueState = new MessageQueueState(messageQueue, processQueue);
}
this.assignedMessageQueueState.put(messageQueue, messageQueueState);
}
}
}
代码比较简单，继续看第二部分消息的拉取
private void updatePullTask(String topic, Set mqNewSet) {
Iterator> it = this.taskTable.entrySet().iterator();
// 移除没有被分配MessageQueue的pullTaskImpl
while (it.hasNext()) {
Map.Entry next = it.next();
if (next.getKey().getTopic().equals(topic)) {
if (!mqNewSet.contains(next.getKey())) {
next.getValue().setCancelled(true);
it.remove();
}
}
}
// 开始拉取任务
startPullTask(mqNewSet);
}
private void startPullTask(Collection mqSet) {
for (MessageQueue messageQueue : mqSet) {
if (!this.taskTable.containsKey(messageQueue)) {
PullTaskImpl pullTask = new PullTaskImpl(messageQueue);
this.taskTable.put(messageQueue, pullTask);
this.scheduledThreadPoolExecutor.schedule(pullTask, 0, TimeUnit.MILLISECONDS);
}
}
}
为每一个messageQueue创建一个PullTaskImpl，然后扔到线程池执行，接下来看看PullTaskImpl的run方法
public void run() {
if (!this.isCancelled()) {
// 当前consumer暂停消费该messageQueue
if (assignedMessageQueue.isPaused(messageQueue)) {
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_WHEN_PAUSE, TimeUnit.MILLISECONDS);
log.debug("Message Queue: {} has been paused!", messageQueue);
return;
}
ProcessQueue processQueue = assignedMessageQueue.getProcessQueue(messageQueue);
// 校验 processQueue
if (null == processQueue || processQueue.isDropped()) {
log.info("The message queue not be able to poll, because it's dropped. group={}, messageQueue={}", defaultLitePullConsumer.getConsumerGroup(), this.messageQueue);
return;
}
// 待获取ConsumeRequest数量 * 一次拉取最多的消息条数 超过限制，进行流控
if (consumeRequestCache.size() * defaultLitePullConsumer.getPullBatchSize() > defaultLitePullConsumer.getPullThresholdForAll()) {
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL, TimeUnit.MILLISECONDS);
if ((consumeRequestFlowControlTimes++ % 1000) == 0)
log.warn("The consume request count exceeds threshold {}, so do flow control, consume request count={}, flowControlTimes={}", consumeRequestCache.size(), consumeRequestFlowControlTimes);
return;
}
// 缓存的消息数量
long cachedMessageCount = processQueue.getMsgCount().get();
// 缓存的消息大小
long cachedMessageSizeInMiB = processQueue.getMsgSize().get() / (1024 * 1024);
// 如果消息数量大于配置，则进行流控，延迟50ms执行消息拉取
if (cachedMessageCount > defaultLitePullConsumer.getPullThresholdForQueue()) {
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL, TimeUnit.MILLISECONDS);
if ((queueFlowControlTimes++ % 1000) == 0) {
log.warn(
"The cached message count exceeds the threshold {}, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, flowControlTimes={}",
defaultLitePullConsumer.getPullThresholdForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, queueFlowControlTimes);
}
return;
}
// 如果消息大小超过配置，也进行流控
if (cachedMessageSizeInMiB > defaultLitePullConsumer.getPullThresholdSizeForQueue()) {
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL, TimeUnit.MILLISECONDS);
if ((queueFlowControlTimes++ % 1000) == 0) {
log.warn(
"The cached message size exceeds the threshold {} MiB, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, flowControlTimes={}",
defaultLitePullConsumer.getPullThresholdSizeForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, queueFlowControlTimes);
}
return;
}
// 单个消息处理队列中最大消息偏移量与最小偏移量的差值触发限流
if (processQueue.getMaxSpan() > defaultLitePullConsumer.getConsumeMaxSpan()) {
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL, TimeUnit.MILLISECONDS);
if ((queueMaxSpanFlowControlTimes++ % 1000) == 0) {
log.warn(
"The queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, flowControlTimes={}",
processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(), queueMaxSpanFlowControlTimes);
}
return;
}
// 如果之前调用过 seek 方法，那么这一次拉取消息会从 seekOffset 开始
// 否则就获取 pullOffset
long offset = nextPullOffset(messageQueue);
long pullDelayTimeMills = 0;
try {
SubscriptionData subscriptionData;
// 启动前已经订阅，则直接获取订阅信息
if (subscriptionType == SubscriptionType.SUBSCRIBE) {
String topic = this.messageQueue.getTopic();
subscriptionData = rebalanceImpl.getSubscriptionInner().get(topic);
}
// 其他两种情况则构造一个订阅信息
else {
String topic = this.messageQueue.getTopic();
subscriptionData = FilterAPI.buildSubscriptionData(defaultLitePullConsumer.getConsumerGroup(),
topic, SubscriptionData.SUB_ALL);
}
// 从broker获取消息
PullResult pullResult = pull(messageQueue, subscriptionData, offset, defaultLitePullConsumer.getPullBatchSize());
switch (pullResult.getPullStatus()) {
case FOUND:
final Object objLock = messageQueueLock.fetchLockObject(messageQueue);
synchronized (objLock) {
// 拉取的消息不为空 并且 期间seekOffset没有变化
if (pullResult.getMsgFoundList() != null && !pullResult.getMsgFoundList().isEmpty() && assignedMessageQueue.getSeekOffset(messageQueue) == -1) {
// 保存拉取到的消息到 processQueue
processQueue.putMessage(pullResult.getMsgFoundList());
// 构建 comsumerRequest，保存到consumeRequestCache中
submitConsumeRequest(new ConsumeRequest(pullResult.getMsgFoundList(), messageQueue, processQueue));
}
}
break;
case OFFSET_ILLEGAL:
log.warn("The pull request offset illegal, {}", pullResult.toString());
break;
default:
break;
}
// 在 assignedMessageQueue 中保存下次开始消费的 offset
updatePullOffset(messageQueue, pullResult.getNextBeginOffset());
} catch (Throwable e) {
pullDelayTimeMills = pullTimeDelayMillsWhenException;
log.error("An error occurred in pull message process.", e);
}
if (!this.isCancelled()) {
scheduledThreadPoolExecutor.schedule(this, pullDelayTimeMills, TimeUnit.MILLISECONDS);
} else {
log.warn("The Pull Task is cancelled after doPullTask, {}", messageQueue);
}
}
}
主要做了这么几件事情
- 判断是否需要流控，也就是延迟执行消息拉取
- 向broker发起请求拉取消息
- 处理从broker的返回
第一件事和第三件事都比较简单，这里就接着看是如何从Broker拉取消息的
private PullResult pull(MessageQueue mq, SubscriptionData subscriptionData, long offset, int maxNums)
throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
return pull(mq, subscriptionData, offset, maxNums, this.defaultLitePullConsumer.getConsumerPullTimeoutMillis());
}
private PullResult pull(MessageQueue mq, SubscriptionData subscriptionData, long offset, int maxNums, long timeout)
throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
return this.pullSyncImpl(mq, subscriptionData, offset, maxNums, true, timeout);
}
private PullResult pullSyncImpl(MessageQueue mq, SubscriptionData subscriptionData, long offset, int maxNums,
boolean block, long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
if (null == mq) {
throw new MQClientException("mq is null", null);
}
if (offset < 0) {
throw new MQClientException("offset < 0", null);
}
if (maxNums <= 0) {
throw new MQClientException("maxNums <= 0", null);
}
// 设置一些默认值：
// commitOffset=false：不提交offset
// block=true：如果队列没有消息，则挂起等待
// subscription=true：是否携带订阅信息
// classFilter和litePull应该是预留字段，暂时没有作用
int sysFlag = PullSysFlag.buildSysFlag(false, block, true, false, true);
// 判断超时时间
long timeoutMillis = block ? this.defaultLitePullConsumer.getConsumerTimeoutMillisWhenSuspend() : timeout;
boolean isTagType = ExpressionType.isTagType(subscriptionData.getExpressionType());
// 发起网络请求，从 broker 拉取消息
PullResult pullResult = this.pullAPIWrapper.pullKernelImpl(
mq,
subscriptionData.getSubString(),
subscriptionData.getExpressionType(),
isTagType ? 0L : subscriptionData.getSubVersion(),
offset,
maxNums,
sysFlag,
0,
this.defaultLitePullConsumer.getBrokerSuspendMaxTimeMillis(),
timeoutMillis,
CommunicationMode.SYNC,
null
);
// 处理 pullResult
this.pullAPIWrapper.processPullResult(mq, pullResult, subscriptionData);
return pullResult;
}
这里就是发起请求和处理请求结果，继续看消息的发送
public PullResult pullKernelImpl(
final MessageQueue mq, // 消息消费队列
final String subExpression, // 消息订阅子模式subscribe( topicName, "模式")
final String expressionType,
final long subVersion, // 版本
final long offset, // pullRequest.getNextOffset()
final int maxNums, // defaultMQPushConsumer.getPullBatchSize()
final int sysFlag, // 系统标记，FLAG_COMMIT_OFFSET FLAG_SUSPEND FLAG_SUBSCRIPTION FLAG_CLASS_FILTER
final long commitOffset, // 当前消息队列 commitlog日志中当前的最新偏移量（内存中）
final long brokerSuspendMaxTimeMillis, // 允许的broker 暂停的时间，毫秒为单位，默认为15s
final long timeoutMillis, // 超时时间,默认为30s
final CommunicationMode communicationMode, // SYNC ASYNC ONEWAY
final PullCallback pullCallback // pull 回调
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
// 根据MQ的Broker信息获取查找Broker信息，封装成FindBrokerResult
FindBrokerResult findBrokerResult =
this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
this.recalculatePullFromWhichNode(mq), false);
// 如果本地找不到，就从 nameServer 拉取节点信息，然后再重新获取
if (null == findBrokerResult) {
this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic());
findBrokerResult =
this.mQClientFactory.findBrokerAddressInSubscribe(mq.getBrokerName(),
this.recalculatePullFromWhichNode(mq), false);
}
if (findBrokerResult != null) {
{
// check version
if (!ExpressionType.isTagType(expressionType)
&& findBrokerResult.getBrokerVersion() < MQVersion.Version.V4_1_0_SNAPSHOT.ordinal()) {
throw new MQClientException("The broker[" + mq.getBrokerName() + ", "
+ findBrokerResult.getBrokerVersion() + "] does not upgrade to support for filter message by " + expressionType, null);
}
}
int sysFlagInner = sysFlag;
// 如果拉取消息的是slave broker，不进行消息确认
if (findBrokerResult.isSlave()) {
sysFlagInner = PullSysFlag.clearCommitOffsetFlag(sysFlagInner);
}
PullMessageRequestHeader requestHeader = new PullMessageRequestHeader();
requestHeader.setConsumerGroup(this.consumerGroup);
requestHeader.setTopic(mq.getTopic());
requestHeader.setQueueId(mq.getQueueId());
requestHeader.setQueueOffset(offset);
requestHeader.setMaxMsgNums(maxNums);
requestHeader.setSysFlag(sysFlagInner);
requestHeader.setCommitOffset(commitOffset);
requestHeader.setSuspendTimeoutMillis(brokerSuspendMaxTimeMillis);
requestHeader.setSubscription(subExpression);
requestHeader.setSubVersion(subVersion);
requestHeader.setExpressionType(expressionType);
String brokerAddr = findBrokerResult.getBrokerAddr();
if (PullSysFlag.hasClassFilterFlag(sysFlagInner)) {
brokerAddr = computePullFromWhichFilterServer(mq.getTopic(), brokerAddr);
}
// 然后通过网络去 拉取具体的消息
PullResult pullResult = this.mQClientFactory.getMQClientAPIImpl().pullMessage(
brokerAddr,
requestHeader,
timeoutMillis,
communicationMode,
pullCallback);
return pullResult;
}
throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}
比较简单，然后看一下请求结果的处理
public PullResult processPullResult(final MessageQueue mq, final PullResult pullResult,
final SubscriptionData subscriptionData) {
PullResultExt pullResultExt = (PullResultExt) pullResult;
// 更新消息队列拉取消息 Broker 编号的映射
this.updatePullFromWhichNode(mq, pullResultExt.getSuggestWhichBrokerId());
// 如果成功拉取到消息
if (PullStatus.FOUND == pullResult.getPullStatus()) {
// 解析消息
ByteBuffer byteBuffer = ByteBuffer.wrap(pullResultExt.getMessageBinary());
List msgList = MessageDecoder.decodes(byteBuffer);
// 根据订阅信息消息 tag 匹配合适消息，这里会过滤掉没有订阅的 tag 消息
List msgListFilterAgain = msgList;
if (!subscriptionData.getTagsSet().isEmpty() && !subscriptionData.isClassFilterMode()) {
msgListFilterAgain = new ArrayList(msgList.size());
for (MessageExt msg : msgList) {
if (msg.getTags() != null) {
if (subscriptionData.getTagsSet().contains(msg.getTags())) {
msgListFilterAgain.add(msg);
}
}
}
}
// Hook
if (this.hasHook()) {
FilterMessageContext filterMessageContext = new FilterMessageContext();
filterMessageContext.setUnitMode(unitMode);
filterMessageContext.setMsgList(msgListFilterAgain);
this.executeHook(filterMessageContext);
}
// 设置消息队列当前最小/最大位置到消息拓展字段
for (MessageExt msg : msgListFilterAgain) {
String traFlag = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
if (Boolean.parseBoolean(traFlag)) {
msg.setTransactionId(msg.getProperty(MessageConst.PROPERTY_UNIQ_CLIENT_MESSAGE_ID_KEYIDX));
}
MessageAccessor.putProperty(msg, MessageConst.PROPERTY_MIN_OFFSET,
Long.toString(pullResult.getMinOffset()));
MessageAccessor.putProperty(msg, MessageConst.PROPERTY_MAX_OFFSET,
Long.toString(pullResult.getMaxOffset()));
msg.setBrokerName(mq.getBrokerName());
}
// 设置消息列表
pullResultExt.setMsgFoundList(msgListFilterAgain);
}
// 清空消息二进制数组
pullResultExt.setMessageBinary(null);
return pullResult;
}
消息解码以及过滤消息
到此MessageQueueListener做的事情就分析完了，总结下就是
- 从broker拉取消息
- 拉取到的消息保存在processQueue中
- 构建 comsumerRequest，保存到consumeRequestCache中。更新pullOffset
回到LitePullConsumerSubscribe中，接下来就是调用DefaultLitePullConsumer的poll方法获取消息
public List poll() {
return defaultLitePullConsumerImpl.poll(this.getPollTimeoutMillis());
}
public synchronized List poll(long timeout) {
try {
checkServiceState();
if (timeout < 0)
throw new IllegalArgumentException("Timeout must not be negative");
if (defaultLitePullConsumer.isAutoCommit()) {
// 指的是保存 offset 到 offsetStore
// 如果是广播模式则还会持久化到本地文件，集群模式下什么也不会做
maybeAutoCommit();
}
long endTime = System.currentTimeMillis() + timeout;
// 这里获取到的就是PullTaskImpl放入consumeRequestCache中的，拉取消息结果
ConsumeRequest consumeRequest = consumeRequestCache.poll(endTime - System.currentTimeMillis(), TimeUnit.MILLISECONDS);
if (endTime - System.currentTimeMillis() > 0) {
while (consumeRequest != null && consumeRequest.getProcessQueue().isDropped()) {
consumeRequest = consumeRequestCache.poll(endTime - System.currentTimeMillis(), TimeUnit.MILLISECONDS);
if (endTime - System.currentTimeMillis() <= 0)
break;
}
}
if (consumeRequest != null && !consumeRequest.getProcessQueue().isDropped()) {
List messages = consumeRequest.getMessageExts();
// 移除在processQueue中缓存的消息
long offset = consumeRequest.getProcessQueue().removeMessage(messages);
// 更新consumerOffset
assignedMessageQueue.updateConsumeOffset(consumeRequest.getMessageQueue(), offset);
//If namespace not null , reset Topic without namespace.
this.resetTopic(messages);
return messages;
}
} catch (InterruptedException ignore) {
}
return Collections.emptyList();
}
到此，DefaultLitePullConsumer的消息消费过程就分析完了