rocketmq 启动_RocketMQ启动与消息发送过程分析

RocketMQ是一款优秀的国产开源消息队列中间件，要了解消息队列关键在了解收发消息，今天我们就来分析一下这款优秀的消息中间件是如何发送消息的。

注：本文并不涉及RPC通信细节，而是从MQ角度来分析启动和消息发送。

准备工作

通过Quick Start了解产品

如何是首次听过RocketMQ，建议通过官网对RocketMQ有一个直观的认识，地址如下：

https://rocketmq.apache.org/docs/quick-start/

动手搭建一下环境，对产品有一个直观感性的认识。

通过一些Example学习了解如何使用，比如：

https://rocketmq.apache.org/docs/simple-example/

了解产品核心概念和特性

核心概念：

https://rocketmq.apache.org/docs/core-concept/

产品特性，比如事务消息：

https://rocketmq.apache.org/docs/transaction-example/

源码与工程模块分析

分析之前需提前下载好源码，地址如下：

https://github.com/apache/rocketmq.git

截止2020-10-06，RocketMQ的发布版本是4.7.1。

RocketMQ工程模块划分清晰，如下图所示：

核心模块在（即主要实现在）：client、broker、remoting、namesrv、store、logging。其中client就包含消息收发实现。

一款优秀的开源软件少不了完备的单元测试，如下图所示：

事实上，RocketMQ也确实包含了大量的测试用例，前期可能会花费大量时间去编写测试用例，看似浪费了大把的时间，实际上回节省出非常多后期联调测试、集成测试以及上线后出问题解决问题的时间，并且能有效降级线上事故的概率，因此这个时间投入是划得来的（在日常开发中，我们也很有必要多写单元测试，做到50%以上的覆盖率）。

测试用例分析

发送消息完备的单元测试用例在org.apache.rocketmq.client.producer.DefaultMQProducerTest，我们来分析一下这个测试类包含的测试用例。如下

这时一个典型的单元测试类结构，有经验的同学大体可以猜测到init和terminate是用于单元测试开始前的初始化准备和测试完成后的销毁代码。而以testSendMessage就是发送消息的各种测试场景。

启动过程

启动过程从org.apache.rocketmq.client.producer.DefaultMQProducerTest#init方法开始分析

@Before
public void init() throws Exception {
    String producerGroupTemp = producerGroupPrefix + System.currentTimeMillis();
    producer = new DefaultMQProducer(producerGroupTemp);// 默认消息发送者
    producer.setNamesrvAddr("127.0.0.1:9876"); // 设置name server，name server是全局配置中心，可以和zk类比
    producer.setCompressMsgBodyOverHowmuch(16); // 消息压缩阈值，这里设置的是16k


    message = new Message(topic, new byte[] {'a'}); // 构造测试消息
    zeroMsg = new Message(topic, new byte[] {});// 构造空消息
    bigMessage = new Message(topic, "This is a very huge message!".getBytes());// 构造大消息


 // 启动生产者
    producer.start();


 // producer注入mQClientFactory
    Field field = DefaultMQProducerImpl.class.getDeclaredField("mQClientFactory");
    field.setAccessible(true);
    field.set(producer.getDefaultMQProducerImpl(), mQClientFactory);


 // mQClientFactory注入mQClientAPIImpl实例
    field = MQClientInstance.class.getDeclaredField("mQClientAPIImpl");
    field.setAccessible(true);
    field.set(mQClientFactory, mQClientAPIImpl);


 // 生产者注册到生产者组中
    producer.getDefaultMQProducerImpl().getmQClientFactory().registerProducer(producerGroupTemp, producer.getDefaultMQProducerImpl());


 // mock callRealMethod
 when(mQClientAPIImpl.sendMessage(anyString(), anyString(), any(Message.class), any(SendMessageRequestHeader.class), anyLong(), any(CommunicationMode.class),
 nullable(SendMessageContext.class), any(DefaultMQProducerImpl.class))).thenCallRealMethod();


 // mock createSendResult
 when(mQClientAPIImpl.sendMessage(anyString(), anyString(), any(Message.class), any(SendMessageRequestHeader.class), anyLong(), any(CommunicationMode.class),
 nullable(SendCallback.class), nullable(TopicPublishInfo.class), nullable(MQClientInstance.class), anyInt(), nullable(SendMessageContext.class), any(DefaultMQProducerImpl.class)))
        .thenReturn(createSendResult(SendStatus.SEND_OK));
}

主要流程就是实例化MQProducer、注入MQClient、调用producer.start()。从中我们可以了解到Producer的默认实现类是org.apache.rocketmq.client.producer.DefaultMQProducer，这个类负责完成消息的发送，start就是启动方法。

在开始分析org.apache.rocketmq.client.producer.DefaultMQProducer#start之前，我们有必要简单了解以下DefaultMQProducer类结构，简单浏览过后你会发现DefaultMQProducer类是一个门面，类结构比较简单，真正的实现类是org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl。

DefaultMQProducer#start代码如下：

public void start() throws MQClientException {
 this.setProducerGroup(withNamespace(this.producerGroup));
 this.defaultMQProducerImpl.start();
 if (null != traceDispatcher) {
 try {
            traceDispatcher.start(this.getNamesrvAddr(), this.getAccessChannel());
        } catch (MQClientException e) {
            log.warn("trace dispatcher start failed ", e);
        }
    }
}

可说的不多，我们继续看下一层DefaultMQProducerImpl#start，代码如下：

public void start() throws MQClientException {
 this.start(true);
}


public void start(final boolean startFactory) throws MQClientException {
 switch (this.serviceState) {
 // 只有服务状态为刚刚创建时才能启动
 case CREATE_JUST:
 this.serviceState = ServiceState.START_FAILED;


 // 检查配置是否正确
 this.checkConfig();


 if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) {
 this.defaultMQProducer.changeInstanceNameToPID();
            }


 // 获取mQClientFactory，单例
 this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQProducer, rpcHook);


 // 注册Producer
 boolean registerOK = mQClientFactory.registerProducer(this.defaultMQProducer.getProducerGroup(), this);
 if (!registerOK) {
 this.serviceState = ServiceState.CREATE_JUST;
 throw new MQClientException("The producer group[" + this.defaultMQProducer.getProducerGroup()
                    + "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
 null);
            }


 // topic PublishInfo
 this.topicPublishInfoTable.put(this.defaultMQProducer.getCreateTopicKey(), new TopicPublishInfo());
 
 // 启动mQClientFactory
 if (startFactory) {
 mQClientFactory.start();
            }


 log.info("the producer [{}] start OK. sendMessageWithVIPChannel={}", this.defaultMQProducer.getProducerGroup(),
 this.defaultMQProducer.isSendMessageWithVIPChannel());
 this.serviceState = ServiceState.RUNNING;
 break;
 case RUNNING:
 case START_FAILED:
 case SHUTDOWN_ALREADY:
 throw new MQClientException("The producer service state not OK, maybe started once, "
                + this.serviceState
                + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
 null);
 default:
 break;
    }


 // 给所有Brokers发送心跳
 this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();


 // 启动定时任务
 this.timer.scheduleAtFixedRate(new TimerTask() {
 @Override
 public void run() {
 try {
                RequestFutureTable.scanExpiredRequest();
            } catch (Throwable e) {
                log.error("scan RequestFutureTable exception", e);
            }
        }
    }, 1000 * 3, 1000);
}

以上，就是Producer启动的主流程，简单说就是：检查配置、获取mQClientFactory并注册自己、启动mQClientFactory、启动完成后最后向所有Broker发送心跳，并创建定时任务。以上代码涉及状态模式和单例。

为了完全弄清楚启动过程，我们有必要继续往下更进，来看下org.apache.rocketmq.client.impl.factory.MQClientInstance#start的实现过程：

public void start() throws MQClientException {


 synchronized (this) {
 switch (this.serviceState) {
 case CREATE_JUST:
 this.serviceState = ServiceState.START_FAILED;


 // 检查name server
 if (null == this.clientConfig.getNamesrvAddr()) {
 this.mQClientAPIImpl.fetchNameServerAddr();
                }


 // 启动request-response channel
 this.mQClientAPIImpl.start();


 // 启动多种调度任务
 this.startScheduledTask();


 // 启动pull message service
 this.pullMessageService.start();


 // 启动负载均衡service
 this.rebalanceService.start();


 // 再次启动DefaultMQProducer
 this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
 log.info("the client factory [{}] start OK", this.clientId);
 this.serviceState = ServiceState.RUNNING;
 break;
 case START_FAILED:
 throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
 default:
 break;
        }
    }
}

到这一层其实分析得差不多了，如果继续往下跟org.apache.rocketmq.remoting.netty.NettyRemotingClient#start，那就是rpc通信底层实现细节了，而rpc底层通信这块内容还不少（涉及通信协议、编解码、序列化协议、网络通信IO），我担心你陷入到这个细节无法自拔。所以源码阅读也是有方法技巧的，如果可能，借助官网资料尽可能了解原理本身，然后带着问题和答案到源码中寻找实现，这样效果可能会更好。

启动源码走读就到这，回顾整个启动过程，主要经过了DefaultMQProducer、DefaultMQProducerImpl、MQClientInstance、MQClientAPIImpl、NettyRemotingClient这几个类，了解各自的职责很有必要，这对于以后遇到问题调试代码很有帮助。

• DefaultMQProducer：Producer门面类，影藏了Producer实现复杂性。
• DefaultMQProducerImpl：Producer 的内部实现类， Producer发消息业务逻辑在这个类中。
• MQClientInstance：封装了客户端通用业务逻辑，无论是 Producer 还是 Consumer，都需要与服务端交互时，都需要调用这个类中的方法；
• MQClientAPIImpl：这个类中封装了客户端服务端的 RPC，对调用者隐藏了真正网络通信部分的具体实现；
• NettyRemotingClient：RocketMQ 各进程之间网络通信的底层实现类。

由于启动过程都是单向的，并不涉及复杂的类与类之间的交互，这里就省略了类的调用时序，简单表示为DefaultMQProducer.start->DefaultMQProducer.start->MQClientInstance.start->MQClientAPIImpl.start->NettyRemoteClient.start。

发送消息过程

按照上面启动流程的分析方式，我们开始分析一下发送消息流程。首先是org.apache.rocketmq.client.producer.DefaultMQProducer

public SendResult send(
    Message msg) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 // 消息结构体合法性校验
    Validators.checkMessage(msg, this);
 // 设置topic
    msg.setTopic(withNamespace(msg.getTopic()));
 // 委派具体实现发送消息
 return this.defaultMQProducerImpl.send(msg);
}

org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl#send

public SendResult send(
    Message msg) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return send(msg, this.defaultMQProducer.getSendMsgTimeout());
}


public SendResult send(Message msg,
    long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
 return this.sendDefaultImpl(msg, CommunicationMode.SYNC, null, timeout);
}


private SendResult sendKernelImpl(final Message msg,
 final MessageQueue mq,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
    long beginStartTime = System.currentTimeMillis();


 // 通过brokerName获取brokerAddress
    String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
 if (null == brokerAddr) {
        tryToFindTopicPublishInfo(mq.getTopic());
        brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
    }


    SendMessageContext context = null;
 if (brokerAddr != null) {
        brokerAddr = MixAll.brokerVIPChannel(this.defaultMQProducer.isSendMessageWithVIPChannel(), brokerAddr);


        byte[] prevBody = msg.getBody();
 try {
 //for MessageBatch,ID has been set in the generating process
 // 消息类型若不是MessageBatch，则需要设置unique id
 if (!(msg instanceof MessageBatch)) {
                MessageClientIDSetter.setUniqID(msg);
            }


 // 带namespace topic
            boolean topicWithNamespace = false;
 if (null != this.mQClientFactory.getClientConfig().getNamespace()) {
                msg.setInstanceId(this.mQClientFactory.getClientConfig().getNamespace());
                topicWithNamespace = true;
            }


 // 压缩消息设置
            int sysFlag = 0;
            boolean msgBodyCompressed = false;
 if (this.tryToCompressMessage(msg)) {
                sysFlag |= MessageSysFlag.COMPRESSED_FLAG;
                msgBodyCompressed = true;
            }


 // 是否是事务消息
 final String tranMsg = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (tranMsg != null && Boolean.parseBoolean(tranMsg)) {
                sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE;
            }
 
 // forbidden hook
 if (hasCheckForbiddenHook()) {
                CheckForbiddenContext checkForbiddenContext = new CheckForbiddenContext();
                checkForbiddenContext.setNameSrvAddr(this.defaultMQProducer.getNamesrvAddr());
                checkForbiddenContext.setGroup(this.defaultMQProducer.getProducerGroup());
                checkForbiddenContext.setCommunicationMode(communicationMode);
                checkForbiddenContext.setBrokerAddr(brokerAddr);
                checkForbiddenContext.setMessage(msg);
                checkForbiddenContext.setMq(mq);
                checkForbiddenContext.setUnitMode(this.isUnitMode());
 this.executeCheckForbiddenHook(checkForbiddenContext);
            }
 
 // send message hook
 if (this.hasSendMessageHook()) {
                context = new SendMessageContext();
                context.setProducer(this);
                context.setProducerGroup(this.defaultMQProducer.getProducerGroup());
                context.setCommunicationMode(communicationMode);
                context.setBornHost(this.defaultMQProducer.getClientIP());
                context.setBrokerAddr(brokerAddr);
                context.setMessage(msg);
                context.setMq(mq);
                context.setNamespace(this.defaultMQProducer.getNamespace());
                String isTrans = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
 if (isTrans != null && isTrans.equals("true")) {
                    context.setMsgType(MessageType.Trans_Msg_Half);
                }


 if (msg.getProperty("__STARTDELIVERTIME") != null || msg.getProperty(MessageConst.PROPERTY_DELAY_TIME_LEVEL) != null) {
                    context.setMsgType(MessageType.Delay_Msg);
                }
 this.executeSendMessageHookBefore(context);
            }
 
 // 构造sendMessageRequestHeader
            SendMessageRequestHeader requestHeader = new SendMessageRequestHeader();
            requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup());
            requestHeader.setTopic(msg.getTopic());
            requestHeader.setDefaultTopic(this.defaultMQProducer.getCreateTopicKey());
            requestHeader.setDefaultTopicQueueNums(this.defaultMQProducer.getDefaultTopicQueueNums());
            requestHeader.setQueueId(mq.getQueueId());
            requestHeader.setSysFlag(sysFlag);
            requestHeader.setBornTimestamp(System.currentTimeMillis());
            requestHeader.setFlag(msg.getFlag());
            requestHeader.setProperties(MessageDecoder.messageProperties2String(msg.getProperties()));
            requestHeader.setReconsumeTimes(0);
            requestHeader.setUnitMode(this.isUnitMode());
            requestHeader.setBatch(msg instanceof MessageBatch);
 if (requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                String reconsumeTimes = MessageAccessor.getReconsumeTime(msg);
 if (reconsumeTimes != null) {
                    requestHeader.setReconsumeTimes(Integer.valueOf(reconsumeTimes));
                    MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_RECONSUME_TIME);
                }


                String maxReconsumeTimes = MessageAccessor.getMaxReconsumeTimes(msg);
 if (maxReconsumeTimes != null) {
                    requestHeader.setMaxReconsumeTimes(Integer.valueOf(maxReconsumeTimes));
                    MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_MAX_RECONSUME_TIMES);
                }
            }


            SendResult sendResult = null;
            switch (communicationMode) {
                case ASYNC:
                    Message tmpMessage = msg;
                    boolean messageCloned = false;
 if (msgBodyCompressed) {
 //If msg body was compressed, msgbody should be reset using prevBody.
 //Clone new message using commpressed message body and recover origin massage.
 //Fix bug:https://github.com/apache/rocketmq-externals/issues/66
                        tmpMessage = MessageAccessor.cloneMessage(msg);
                        messageCloned = true;
                        msg.setBody(prevBody);
                    }


 if (topicWithNamespace) {
 if (!messageCloned) {
                            tmpMessage = MessageAccessor.cloneMessage(msg);
                            messageCloned = true;
                        }
                        msg.setTopic(NamespaceUtil.withoutNamespace(msg.getTopic(), this.defaultMQProducer.getNamespace()));
                    }


                    long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
                    }
 // 发送消息
                    sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
                        brokerAddr,
                        mq.getBrokerName(),
                        tmpMessage,
                        requestHeader,
                        timeout - costTimeAsync,
                        communicationMode,
                        sendCallback,
                        topicPublishInfo,
 this.mQClientFactory,
 this.defaultMQProducer.getRetryTimesWhenSendAsyncFailed(),
                        context,
 this);
 break;
                case ONEWAY:
                case SYNC:
                    long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeout < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
                    }
 // 发送消息
                    sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
                        brokerAddr,
                        mq.getBrokerName(),
                        msg,
                        requestHeader,
                        timeout - costTimeSync,
                        communicationMode,
                        context,
 this);
 break;
 default:
                    assert false;
 break;
            }


 if (this.hasSendMessageHook()) {
                context.setSendResult(sendResult);
 this.executeSendMessageHookAfter(context);
            }


 return sendResult;
        } catch (RemotingException e) {
 if (this.hasSendMessageHook()) {
                context.setException(e);
 this.executeSendMessageHookAfter(context);
            }
 throw e;
        } catch (MQBrokerException e) {
 if (this.hasSendMessageHook()) {
                context.setException(e);
 this.executeSendMessageHookAfter(context);
            }
 throw e;
        } catch (InterruptedException e) {
 if (this.hasSendMessageHook()) {
                context.setException(e);
 this.executeSendMessageHookAfter(context);
            }
 throw e;
        } finally {
            msg.setBody(prevBody);
            msg.setTopic(NamespaceUtil.withoutNamespace(msg.getTopic(), this.defaultMQProducer.getNamespace()));
        }
    }


 throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}


接着调用org.apache.rocketmq.client.impl.MQClientAPIImpl#sendMessage发送消息：
public SendResult sendMessage(
 final String addr,
 final String brokerName,
 final Message msg,
 final SendMessageRequestHeader requestHeader,
 final long timeoutMillis,
 final CommunicationMode communicationMode,
 final SendCallback sendCallback,
 final TopicPublishInfo topicPublishInfo,
 final MQClientInstance instance,
 final int retryTimesWhenSendFailed,
 final SendMessageContext context,
 final DefaultMQProducerImpl producer
) throws RemotingException, MQBrokerException, InterruptedException {
 long beginStartTime = System.currentTimeMillis();
    RemotingCommand request = null;
    String msgType = msg.getProperty(MessageConst.PROPERTY_MESSAGE_TYPE);
 boolean isReply = msgType != null && msgType.equals(MixAll.REPLY_MESSAGE_FLAG);
 if (isReply) {
 if (sendSmartMsg) {
            SendMessageRequestHeaderV2 requestHeaderV2 = SendMessageRequestHeaderV2.createSendMessageRequestHeaderV2(requestHeader);
            request = RemotingCommand.createRequestCommand(RequestCode.SEND_REPLY_MESSAGE_V2, requestHeaderV2);
        } else {
            request = RemotingCommand.createRequestCommand(RequestCode.SEND_REPLY_MESSAGE, requestHeader);
        }
    } else {
 if (sendSmartMsg || msg instanceof MessageBatch) {
            SendMessageRequestHeaderV2 requestHeaderV2 = SendMessageRequestHeaderV2.createSendMessageRequestHeaderV2(requestHeader);
            request = RemotingCommand.createRequestCommand(msg instanceof MessageBatch ? RequestCode.SEND_BATCH_MESSAGE : RequestCode.SEND_MESSAGE_V2, requestHeaderV2);
        } else {
            request = RemotingCommand.createRequestCommand(RequestCode.SEND_MESSAGE, requestHeader);
        }
    }
    request.setBody(msg.getBody());


 switch (communicationMode) {
 case ONEWAY:
 this.remotingClient.invokeOneway(addr, request, timeoutMillis);
 return null;
 case ASYNC:
 final AtomicInteger times = new AtomicInteger();
 long costTimeAsync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeAsync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
            }
 this.sendMessageAsync(addr, brokerName, msg, timeoutMillis - costTimeAsync, request, sendCallback, topicPublishInfo, instance,
                retryTimesWhenSendFailed, times, context, producer);
 return null;
 case SYNC:
 long costTimeSync = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTimeSync) {
 throw new RemotingTooMuchRequestException("sendMessage call timeout");
            }
 return this.sendMessageSync(addr, brokerName, msg, timeoutMillis - costTimeSync, request);
 default:
 assert false;
 break;
    }


 return null;
}

到这，同步发送代码分析差不多了，继续往下更近也是rpc底层通信的实现细节，以后我们有机会再分析rpc通信这块内容。

整个调用链为：DefaultMQProducer.send-> DefaultMQProducerImpl.send->MQClientAPIImpl.sendMessage->RemotingClient#invokeOneway or RemotingClient#invokeAsync

异步发送主要差别是是否使用了ExecutorService提交异步任务。

总结

本文分析了RocketMQ启动与消息发送两个过程，读者朋友可透过本文了解开源软件源码的分析方式，这里做个总结：

1. 首先，通过文档了解原理。
2. 其次，带着问题和答案开始分析代码实现。
3. 然后，通过工程结构定位模块，并找到相关实现所在位置。
4. 再次，通过单元测试找到入库开始走读代码、分析业务流程。
5. 最后，分析类调用时序，分析类的职责。

由于RPC通信涉及编解码、序列化和网络通信IO模型，因此本文并不打算详讲RPC通信细节。希望对你有所帮助。

以上，转载请注明来源。