RocketMQ源码分析之Message消费与拉取（下Consume的拉取消费）
各组件大致功能图
PushConsumer订阅
DefaultPushConsumerImpl#subscrible
首先会创建订阅数据，在里面会根据topic、和订阅表达式将订阅信息放入到SubscriptionInner中，然后通过加分布式锁的方式进行心跳同步Consumer信息到所有的Broker
1: public void subscribe(String topic, String subExpression) throws MQClientException {
2: try {
3: // 创建订阅数据
4: SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPushConsumer.getConsumerGroup(), //
5: topic, subExpression);
6: this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
7: // 通过心跳同步Consumer信息到Broker
8: if (this.mQClientFactory != null) {
9: this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
10: }
11: } catch (Exception e) {
12: throw new MQClientException("subscription exception", e);
13: }
14: }PushConsumer消息队列分配（Reblance）
在消费队列负载均衡服务中，负责分配当前Consumer可消费的消息队列
当等待超时（20s）或者PushCo启动或者有Consum加入移除的时候，会引MQClientInstance#doRebalance(…) 分配消息队列【该方法中，会遍历当前的Cient下的消费者集合表，并分别让每个MQComsumerInner执行doRelance均衡任务】
MQClientInstance#doRebalance(…)
1: public void doRebalance() {
2: for (Map.Entry entry : this.consumerTable.entrySet()) {
3: MQConsumerInner impl = entry.getValue();
4: if (impl != null) {
5: try {
6: impl.doRebalance();
7: } catch (Throwable e) {
8: log.error("doRebalance exception", e);
9: }
10: }
11: }
12: }RebalanceImpl#rebalanceByTopic(…)
两种模式，一种是广播模式，一种是集群模式，如果是广播模式，则会分配Topic对应的所有的消息队列；如果是集群模式，会分配对应的部分消费队列【内部需要用Collections.sort队消息队列和消费者数组进行排序，这样才能保证各个Consumer顺序一致，并且会根据相应的队列分配策略分配队列，最后会更新Topic对应的消息队列】
1: private void rebalanceByTopic(final String topic, final boolean isOrder) {
2: switch (messageModel) {
3: case BROADCASTING: {
4: Set mqSet = this.topicSubscribeInfoTable.get(topic);
5: if (mqSet != null) {
6: boolean changed = this.updateProcessQueueTableInRebalance(topic, mqSet, isOrder);
7: if (changed) {
8: this.messageQueueChanged(topic, mqSet, mqSet);
9: log.info("messageQueueChanged {} {} {} {}", //
10: consumerGroup, //
11: topic, //
12: mqSet, //
13: mqSet);
14: }
15: } else {
16: log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
17: }
18: break;
19: }
20: case CLUSTERING: {
21: // 获取 topic 对应的 队列 和 consumer信息
22: Set mqSet = this.topicSubscribeInfoTable.get(topic);
23: List cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
24: if (null == mqSet) {
25: if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
26: log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
27: }
28: }
29:
30: if (null == cidAll) {
31: log.warn("doRebalance, {} {}, get consumer id list failed", consumerGroup, topic);
32: }
33:
34: if (mqSet != null && cidAll != null) {
35: // 排序 消息队列 和 消费者数组。因为是在Client进行分配队列，排序后，各Client的顺序才能保持一致。
36: List mqAll = new ArrayList<>();
37: mqAll.addAll(mqSet);
38:
39: Collections.sort(mqAll);
40: Collections.sort(cidAll);
41:
42: AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
43:
44: // 根据 队列分配策略 分配消息队列
45: List allocateResult;
46: try {
47: allocateResult = strategy.allocate(//
48: this.consumerGroup, //
49: this.mQClientFactory.getClientId(), //
50: mqAll, //
51: cidAll);
52: } catch (Throwable e) {
53: log.error("AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}", strategy.getName(),
54: e);
55: return;
56: }
57:
58: Set allocateResultSet = new HashSet<>();
59: if (allocateResult != null) {
60: allocateResultSet.addAll(allocateResult);
61: }
62:
63: // 更新消息队列
64: boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
65: if (changed) {
66: log.info(
67: "rebalanced result changed. allocateMessageQueueStrategyName={}, group={}, topic={}, clientId={}, mqAllSize={}, cidAllSize={}, rebalanceResultSize={}, rebalanceResultSet={}",
68: strategy.getName(), consumerGroup, topic, this.mQClientFactory.getClientId(), mqSet.size(), cidAll.size(),
69: allocateResultSet.size(), allocateResultSet);
70: this.messageQueueChanged(topic, mqSet, allocateResultSet);
71: }
72: }
73: break;
74: }
75: default:
76: break;
77: }
78: }
79:
80: /**
81: * 当负载均衡时，更新 消息处理队列
82: * - 移除 在processQueueTable && 不存在于 mqSet 里的消息队列
83: * - 增加 不在processQueueTable && 存在于mqSet 里的消息队列
84: *
85: * @param topic Topic
86: * @param mqSet 负载均衡结果后的消息队列数组
87: * @param isOrder 是否顺序
88: * @return 是否变更
89: */
90: private boolean updateProcessQueueTableInRebalance(final String topic, final Set mqSet, final boolean isOrder) {
91: boolean changed = false;
92:
93: // 移除 在processQueueTable && 不存在于 mqSet 里的消息队列
94: Iterator> it = this.processQueueTable.entrySet().iterator();
95: while (it.hasNext()) { // TODO 待读：
96: Entry next = it.next();
97: MessageQueue mq = next.getKey();
98: ProcessQueue pq = next.getValue();
99:
100: if (mq.getTopic().equals(topic)) {
101: if (!mqSet.contains(mq)) { // 不包含的队列
102: pq.setDropped(true);
103: if (this.removeUnnecessaryMessageQueue(mq, pq)) {
104: it.remove();
105: changed = true;
106: log.info("doRebalance, {}, remove unnecessary mq, {}", consumerGroup, mq);
107: }
108: } else if (pq.isPullExpired()) { // 队列拉取超时，进行清理
109: switch (this.consumeType()) {
110: case CONSUME_ACTIVELY:
111: break;
112: case CONSUME_PASSIVELY:
113: pq.setDropped(true);
114: if (this.removeUnnecessaryMessageQueue(mq, pq)) {
115: it.remove();
116: changed = true;
117: log.error("[BUG]doRebalance, {}, remove unnecessary mq, {}, because pull is pause, so try to fixed it",
118: consumerGroup, mq);
119: }
120: break;
121: default:
122: break;
123: }
124: }
125: }
126: }
127:
128: // 增加 不在processQueueTable && 存在于mqSet 里的消息队列。
129: List pullRequestList = new ArrayList<>(); // 拉消息请求数组
130: for (MessageQueue mq : mqSet) {
131: if (!this.processQueueTable.containsKey(mq)) {
132: if (isOrder && !this.lock(mq)) {
133: log.warn("doRebalance, {}, add a new mq failed, {}, because lock failed", consumerGroup, mq);
134: continue;
135: }
136:
137: this.removeDirtyOffset(mq);
138: ProcessQueue pq = new ProcessQueue();
139: long nextOffset = this.computePullFromWhere(mq);
140: if (nextOffset >= 0) {
141: ProcessQueue pre = this.processQueueTable.putIfAbsent(mq, pq);
142: if (pre != null) {
143: log.info("doRebalance, {}, mq already exists, {}", consumerGroup, mq);
144: } else {
145: log.info("doRebalance, {}, add a new mq, {}", consumerGroup, mq);
146: PullRequest pullRequest = new PullRequest();
147: pullRequest.setConsumerGroup(consumerGroup);
148: pullRequest.setNextOffset(nextOffset);
149: pullRequest.setMessageQueue(mq);
150: pullRequest.setProcessQueue(pq);
151: pullRequestList.add(pullRequest);
152: changed = true;
153: }
154: } else {
155: log.warn("doRebalance, {}, add new mq failed, {}", consumerGroup, mq);
156: }
157: }
158: }
159:
160: // 发起消息拉取请求
161: this.dispatchPullRequest(pullRequestList);
162:
163: return changed;
164: }PushConsumer拉取消息
消息拉取服务会不断地从Broker的消息队列中拉取消息，并提交到消费任务到ConsumeMessageService中
public class PullMessageService extends ServiceThread {
@Override
public void run() {
log.info(this.getServiceName() + " service started");
while (!this.isStopped()) {
try {
PullRequest pullRequest = this.pullRequestQueue.take();
if (pullRequest != null) {
this.pullMessage(pullRequest);
}
} catch (InterruptedException e) {
} catch (Exception e) {
log.error("Pull Message Service Run Method exception", e);
}
}
log.info(this.getServiceName() + " service end");
}
}PushConsumer消费消息
ConsumeMessageConcurrentlyService 提交消费请求ConsumeMessageConcurrentlyService#submitConsumeRequest(…)
1: /**
2: * 消费线程池队列
3: */
4: private final BlockingQueue consumeRequestQueue;
5: /**
6: * 消费线程池
7: */
8: private final ThreadPoolExecutor consumeExecutor;
9:
10: public void submitConsumeRequest(//
11: final List msgs, //
12: final ProcessQueue processQueue, //
13: final MessageQueue messageQueue, //
14: final boolean dispatchToConsume) {
15: final int consumeBatchSize = this.defaultMQPushConsumer.getConsumeMessageBatchMaxSize();
16: if (msgs.size() <= consumeBatchSize) { // 提交消息小于批量消息数，直接提交消费请求
17: ConsumeRequest consumeRequest = new ConsumeRequest(msgs, processQueue, messageQueue);
18: try {
19: this.consumeExecutor.submit(consumeRequest);
20: } catch (RejectedExecutionException e) {
21: this.submitConsumeRequestLater(consumeRequest);
22: }
23: } else { // 提交消息大于批量消息数，进行分拆成多个消费请求
24: for (int total = 0; total < msgs.size(); ) {
25: // 计算当前拆分请求包含的消息
26: List msgThis = new ArrayList<>(consumeBatchSize);
27: for (int i = 0; i < consumeBatchSize; i++, total++) {
28: if (total < msgs.size()) {
29: msgThis.add(msgs.get(total));
30: } else {
31: break;
32: }
33: }
34:
35: // 提交拆分消费请求
36: ConsumeRequest consumeRequest = new ConsumeRequest(msgThis, processQueue, messageQueue);
37: try {
38: this.consumeExecutor.submit(consumeRequest);
39: } catch (RejectedExecutionException e) {
40: // 如果被拒绝，则将当前拆分消息+剩余消息提交延迟消费请求。
41: for (; total < msgs.size(); total++) {
42: msgThis.add(msgs.get(total));
43: }
44: this.submitConsumeRequestLater(consumeRequest);
45: }
46: }
47: }
48: }