RocketMQ Broker启动流程解析
RocketMQ中Broker模块主要负责消息的存储、投递和查询以及服务高可用保证，它是RocketMQ中最为核心的模块，下面介绍一下它的启动流程。
启动入口
Broker是通过BrokerStartup的主函数启动的，代码如下：
public static void main(String[] args) {
start(createBrokerController(args));
}先通过函数createBrokerController创建BrokerController对象，然后用start方法启动。创建BrokerController对象的代码如下：
public static BrokerController createBrokerController(String[] args) {
//MQ的版本号
System.setProperty(RemotingCommand.REMOTING_VERSION_KEY, Integer.toString(MQVersion.CURRENT_VERSION));
//设置broker的netty客户端的发送缓冲大小，默认是128 kb
if (null == System.getProperty(NettySystemConfig.COM_ROCKETMQ_REMOTING_SOCKET_SNDBUF_SIZE)) {
NettySystemConfig.socketSndbufSize = 131072;
}
//设置broker的netty客户端的接收缓冲大小，默认是128 kb
if (null == System.getProperty(NettySystemConfig.COM_ROCKETMQ_REMOTING_SOCKET_RCVBUF_SIZE)) {
NettySystemConfig.socketRcvbufSize = 131072;
}
try {
//PackageConflictDetect.detectFastjson();
//命令行选项解析
Options options = ServerUtil.buildCommandlineOptions(new Options());
//解析命令栏中的 mqbroker
commandLine = ServerUtil.parseCmdLine("mqbroker", args, buildCommandlineOptions(options),
new PosixParser());
if (null == commandLine) {
System.exit(-1);
}
//相关配置的存储对象
final BrokerConfig brokerConfig = new BrokerConfig();
final NettyServerConfig nettyServerConfig = new NettyServerConfig();
final NettyClientConfig nettyClientConfig = new NettyClientConfig();
//是否使用TLS （TLS是SSL的升级版本，TLS是SSL的标准化后的产物，有1.0 1.1 1.2三个版本）
nettyClientConfig.setUseTLS(Boolean.parseBoolean(System.getProperty(TLS_ENABLE,
String.valueOf(TlsSystemConfig.tlsMode == TlsMode.ENFORCING))));
//设置netty的服务端监听的端口 10911
nettyServerConfig.setListenPort(10911);
//消息存储相关的配置
final MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
//如果Broker是Slave角色，消息占用内存大小的比率 比默认的40% 还要小 10%
if (BrokerRole.SLAVE == messageStoreConfig.getBrokerRole()) {
int ratio = messageStoreConfig.getAccessMessageInMemoryMaxRatio() - 10;
//设置消息的内存最大占比，如果内存占比超过设定值，那么就进行置换
messageStoreConfig.setAccessMessageInMemoryMaxRatio(ratio);
}
......省略配置读取保存相关代码
//创建BrokerController
final BrokerController controller = new BrokerController(
brokerConfig,
nettyServerConfig,
nettyClientConfig,
messageStoreConfig);
// remember all configs to prevent discard
//缓存额外配置
controller.getConfiguration().registerConfig(properties);
//初始化BrokerController
boolean initResult = controller.initialize();
if (!initResult) {
controller.shutdown();
System.exit(-3);
}
//注册关闭的钩子方法
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
private volatile boolean hasShutdown = false;
private AtomicInteger shutdownTimes = new AtomicInteger(0);
@Override
public void run() {
synchronized (this) {
log.info("Shutdown hook was invoked, {}", this.shutdownTimes.incrementAndGet());
if (!this.hasShutdown) {
this.hasShutdown = true;
long beginTime = System.currentTimeMillis();
//BrokerController的销毁方法
controller.shutdown();
long consumingTimeTotal = System.currentTimeMillis() - beginTime;
log.info("Shutdown hook over, consuming total time(ms): {}", consumingTimeTotal);
}
}
}
}, "ShutdownHook"));
return controller;
} catch (Throwable e) {
e.printStackTrace();
System.exit(-1);
}
return null;
}
函数createBrokerController的主要流程是获取启动参数和命令单中的配置参数以及默认的配置参数进行融合。然后使用这些配置对象创建BrokerController，然后调用初始化方法controller.initialize()，创建一些线程池和对一些模块进行初始化。
Broker分为Master和Slave，brokerId=0表示Master，Slave大于1表示Slave。
Broker在启动的时候会开三个端口
| 端口 | 作用 |
|---|---|
| 10911 | 接收消息推送的端口 |
| 10912 | 高可用的端口 |
| 10909 | 推送消息的vip端口 |
BrokerController的构造方法
BrokerController通过4个配置类来构造，代码如下：
public BrokerController(
final BrokerConfig brokerConfig,
final NettyServerConfig nettyServerConfig,
final NettyClientConfig nettyClientConfig,
final MessageStoreConfig messageStoreConfig
) {
//前面准备的配置信息
this.brokerConfig = brokerConfig;
this.nettyServerConfig = nettyServerConfig;
this.nettyClientConfig = nettyClientConfig;
this.messageStoreConfig = messageStoreConfig;
//Consumer消费进度记录管理类
this.consumerOffsetManager = new ConsumerOffsetManager(this);
//消息topic维度的管理查询类 管理topic和topic相关配置关系
this.topicConfigManager = new TopicConfigManager(this);
//Consumer端使用pull的方式向Broker拉取消息请求的处理类
this.pullMessageProcessor = new PullMessageProcessor(this);
//Consumer端使用pull的方式拉取请求时，保存请求，当有消息到达时进行推送处理类
this.pullRequestHoldService = new PullRequestHoldService(this);
//消息到达Broker的时候的监听回调类，这里会调用到pullRequestHoldService中的方法
this.messageArrivingListener = new NotifyMessageArrivingListener(this.pullRequestHoldService);
//消费者id变化监听器
this.consumerIdsChangeListener = new DefaultConsumerIdsChangeListener(this);
//消费者管理类 按照group进行分组，对消费者的id变化进行监听
this.consumerManager = new ConsumerManager(this.consumerIdsChangeListener);
//消费者的过滤管理类 按照topic进行分类
this.consumerFilterManager = new ConsumerFilterManager(this);
//生产者管理 按照group进行分类
this.producerManager = new ProducerManager();
//心跳连接处理类
this.clientHousekeepingService = new ClientHousekeepingService(this);
//控制台用的
this.broker2Client = new Broker2Client(this);
//订阅关系管理类
this.subscriptionGroupManager = new SubscriptionGroupManager(this);
//broker对外api
this.brokerOuterAPI = new BrokerOuterAPI(nettyClientConfig);
this.filterServerManager = new FilterServerManager(this);
//从broker 同步进度管理类
this.slaveSynchronize = new SlaveSynchronize(this);
//各种线程池的阻塞队列
this.sendThreadPoolQueue = new LinkedBlockingQueue(this.brokerConfig.getSendThreadPoolQueueCapacity());
this.pullThreadPoolQueue = new LinkedBlockingQueue(this.brokerConfig.getPullThreadPoolQueueCapacity());
this.replyThreadPoolQueue = new LinkedBlockingQueue(this.brokerConfig.getReplyThreadPoolQueueCapacity());
this.queryThreadPoolQueue = new LinkedBlockingQueue(this.brokerConfig.getQueryThreadPoolQueueCapacity());
this.clientManagerThreadPoolQueue = new LinkedBlockingQueue(this.brokerConfig.getClientManagerThreadPoolQueueCapacity());
this.consumerManagerThreadPoolQueue = new LinkedBlockingQueue(this.brokerConfig.getConsumerManagerThreadPoolQueueCapacity());
this.heartbeatThreadPoolQueue = new LinkedBlockingQueue(this.brokerConfig.getHeartbeatThreadPoolQueueCapacity());
this.endTransactionThreadPoolQueue = new LinkedBlockingQueue(this.brokerConfig.getEndTransactionPoolQueueCapacity());
this.brokerStatsManager = new BrokerStatsManager(this.brokerConfig.getBrokerClusterName());
this.setStoreHost(new InetSocketAddress(this.getBrokerConfig().getBrokerIP1(), this.getNettyServerConfig().getListenPort()));
this.brokerFastFailure = new BrokerFastFailure(this);
this.configuration = new Configuration(
log,
BrokerPathConfigHelper.getBrokerConfigPath(),
this.brokerConfig, this.nettyServerConfig, this.nettyClientConfig, this.messageStoreConfig
);
}
在创建BrokerController时，会创建各种处理对象，下面列举一下这些核心类的作用
| 字段类 | 作用 |
| ConsumerOffsetManager | Consumer消费者的消费进度记录管理类 |
| TopicConfigManager | 消息topic维度的管理查询类 |
| PullMessageProcessor | Consumer端使用pull的方式向Broker拉取消息请求的处理类 |
| PullRequestHoldService | Consumer端使用push的方式拉取请求时，保存请求，当有消息到达时进行推送处理类。对于push方式消费，会对消费端的请求进行保存，当有消息到达的时候然后进行推送 |
| NotifyMessageArrivingListener | 消息到达Broker的时候的监听回调类，这里会调用到pullRequestHoldService中的notifyMessageArriving方法 |
| DefaultConsumerIdsChangeListener | 消费者id变化监听器,主要监听Consumer的注册和下线的事件 |
| ConsumerManager | 消费者管理类 按照group进行分组，对消费者的id变化进行监听 |
| ConsumerFilterManager | 消费者的过滤管理类 按照topic进行分类 |
| ProducerManager | 生产者管理 按照group进行分类 |
| ClientHousekeepingService | 心跳连接处理类 |
| Broker2Client | 控制台获取Broker信息用 |
| SubscriptionGroupManager | 订阅关系管理类 |
| BrokerOuterAPI | broker对外api |
| SlaveSynchronize | 从broker 同步进度管理类 |
BrokerController的初始化initialize
BrokerController在创建好以后就调用函数initialize进行初始化，代码如下：
public boolean initialize() throws CloneNotSupportedException {
//加载 topic 相关配置，文件地址为 {user.home}/store/config/topics.json
boolean result = this.topicConfigManager.load();
//加载 不同的Consumer消费的进度情况 文件地址为 {user.home}/store/config/consumerOffset.json
result = result && this.consumerOffsetManager.load();
//加载 订阅关系 文件地址 {user.home}/store/config/subscriptionGroup.json
result = result && this.subscriptionGroupManager.load();
//加载 Consumer的过滤信息配置 文件地址 {user.home}/store/config/consumerFilter.json
result = result && this.consumerFilterManager.load();
//如果上述文件加载正常
if (result) {
try {
//创建DefaultMessageStore，
this.messageStore =
new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener,
this.brokerConfig);
//使用的是DLegerCommitLog，则创建DLedgerRoleChangeHandler
if (messageStoreConfig.isEnableDLegerCommitLog()) {
DLedgerRoleChangeHandler roleChangeHandler = new DLedgerRoleChangeHandler(this, (DefaultMessageStore) messageStore);
((DLedgerCommitLog)((DefaultMessageStore) messageStore).getCommitLog()).getdLedgerServer().getdLedgerLeaderElector().addRoleChangeHandler(roleChangeHandler);
}
//Broker的消息统计类
this.brokerStats = new BrokerStats((DefaultMessageStore) this.messageStore);
//load plugin
MessageStorePluginContext context = new MessageStorePluginContext(messageStoreConfig, brokerStatsManager, messageArrivingListener, brokerConfig);
this.messageStore = MessageStoreFactory.build(context, this.messageStore);
this.messageStore.getDispatcherList().addFirst(new CommitLogDispatcherCalcBitMap(this.brokerConfig, this.consumerFilterManager));
} catch (IOException e) {
result = false;
log.error("Failed to initialize", e);
}
}
//加载消息的日志文件，包含CommitLog，ConsumeQueue等
result = result && this.messageStore.load();
//如果上述文件存在一个加载不成功则直接返回
if (result) {
//开启服务端
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.clientHousekeepingService);
NettyServerConfig fastConfig = (NettyServerConfig) this.nettyServerConfig.clone();
fastConfig.setListenPort(nettyServerConfig.getListenPort() - 2);
//再开一个端口为10909 的服务端口，这个端口只给 消息的生产者使用
this.fastRemotingServer = new NettyRemotingServer(fastConfig, this.clientHousekeepingService);
//处理消息生产者发送的生成消息api 相关的线程池
this.sendMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getSendMessageThreadPoolNums(),
this.brokerConfig.getSendMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.sendThreadPoolQueue,
new ThreadFactoryImpl("SendMessageThread_"));
//处理消费者发出的消费消息api 相关的线程池
this.pullMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getPullMessageThreadPoolNums(),
this.brokerConfig.getPullMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.pullThreadPoolQueue,
new ThreadFactoryImpl("PullMessageThread_"));
//处理回复消息api的线程池
this.replyMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getProcessReplyMessageThreadPoolNums(),
this.brokerConfig.getProcessReplyMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.replyThreadPoolQueue,
new ThreadFactoryImpl("ProcessReplyMessageThread_"));
//查询线程
this.queryMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getQueryMessageThreadPoolNums(),
this.brokerConfig.getQueryMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.queryThreadPoolQueue,
new ThreadFactoryImpl("QueryMessageThread_"));
//-------------------------省略部分线程池的创建------------------------------//
//为客户端注册需要处理API指令事件，以及消息发送和消费的回调方法
this.registerProcessor();
//-------------------------省略定时执行任务服务------------------------------//
//初始化事务消息相关的服务
initialTransaction();
//消息轨迹
initialAcl();
//Rpc调用的钩子
initialRpcHooks();
}
private void initialTransaction() {
//加载TransactionalMessageService服务，实现类为TransactionalMessageServiceImpl
this.transactionalMessageService = ServiceProvider.loadClass(ServiceProvider.TRANSACTION_SERVICE_ID, TransactionalMessageService.class);
if (null == this.transactionalMessageService) {
this.transactionalMessageService = new TransactionalMessageServiceImpl(new TransactionalMessageBridge(this, this.getMessageStore()));
log.warn("Load default transaction message hook service: {}", TransactionalMessageServiceImpl.class.getSimpleName());
}
//AbstractTransactionalMessageCheckListener对应的服务类为LogTransactionalMessageCheckListener ，其中实现为空实现
this.transactionalMessageCheckListener = ServiceProvider.loadClass(ServiceProvider.TRANSACTION_LISTENER_ID, AbstractTransactionalMessageCheckListener.class);
if (null == this.transactionalMessageCheckListener) {
this.transactionalMessageCheckListener = new DefaultTransactionalMessageCheckListener();
log.warn("Load default discard message hook service: {}", DefaultTransactionalMessageCheckListener.class.getSimpleName());
}
//设置对应的brokerController到AbstractTransactionalMessageCheckListener中
this.transactionalMessageCheckListener.setBrokerController(this);
//创建TransactionalMessageCheckService，这个服务是周期检查事务的服务，
this.transactionalMessageCheckService = new TransactionalMessageCheckService(this);
}
private void initialAcl() {
if (!this.brokerConfig.isAclEnable()) {
log.info("The broker dose not enable acl");
return;
}
//初始化AccessValidator对应的实现PlainAccessValidator
List accessValidators = ServiceProvider.load(ServiceProvider.ACL_VALIDATOR_ID, AccessValidator.class);
if (accessValidators == null || accessValidators.isEmpty()) {
log.info("The broker dose not load the AccessValidator");
return;
}
//把对应的权限校验加入对应的校验器中
for (AccessValidator accessValidator: accessValidators) {
final AccessValidator validator = accessValidator;
accessValidatorMap.put(validator.getClass(),validator);
this.registerServerRPCHook(new RPCHook() {
@Override
public void doBeforeRequest(String remoteAddr, RemotingCommand request) {
//Do not catch the exception
validator.validate(validator.parse(request, remoteAddr));
}
@Override
public void doAfterResponse(String remoteAddr, RemotingCommand request, RemotingCommand response) {
}
});
}
}
private void initialRpcHooks() {
//加载对应的RPC钩子方法
List rpcHooks = ServiceProvider.load(ServiceProvider.RPC_HOOK_ID, RPCHook.class);
if (rpcHooks == null || rpcHooks.isEmpty()) {
return;
}
for (RPCHook rpcHook: rpcHooks) {
//注册钩子方法
this.registerServerRPCHook(rpcHook);
}
}
initialize方法主要分为下面几个步骤：
1、加载相关的日志文件，包括
topic相关配置，文件地址为 {user.home}/store/config/topics.json；
不同的Consumer消费的进度情况 文件地址为 {user.home}/store/config/consumerOffset.json；
订阅关系 文件地址 {user.home}/store/config/subscriptionGroup.json；
Consumer的过滤信息配置 文件地址 {user.home}/store/config/consumerFilter.json。
2、初始化线程池，包括初始化发送消息线程池、拉取消息线程池、broker管理线程池等
3、注册事件到对应的Broker客户端上，然后会记录对应的API事件和对应线程池封装到一个对象中。
4、启动一些定时任务，这些任务比如记录Broker状态，消费进度持久化等任务。
5、初始化一些服务，比如事务相关（周期检查事务），消息权限校验初始化和Rpc调用钩子相关服务。对应的服务加载方式是Java的SPI方式。
总结
通过前面的分析可知，Broker的启动流程主要是先初始化4个配置文件，然后创建BrokerController对象，再调用initialize进行初始化，最后调用函数start进行启动。流程图如下：
