RocketMQ源码学习四：Broker启动
目录
-
-
-
-
- BrokerStartup的作用
- 源码
-
- DefaultMessageStore初始化
- CommitLog、ConsumeQueue、IndexFile的初始化
- 初始化各种线程池
- 向RemotingServer注册各种处理器
- 启动各种定时任务
- 其他初始化
- 总结
-
-
-
BrokerStartup的作用
启动Broker服务器实例。它负责加载配置文件、初始化各种组件和服务，并启动消息存储、消息消费和消息索引等功能。通过运行BrokerStartup类，您可以启动RocketMQ Broker，使其能够接收和处理消息。
源码
public static void main(String[] args) {
start(createBrokerController(args));
}
public static BrokerController start(BrokerController controller) {
try {
/**
* broker启动类，里边封装了各个组件的初始化以及启动逻辑
*/
controller.start();
String tip = "The broker[" + controller.getBrokerConfig().getBrokerName() + ", "
+ controller.getBrokerAddr() + "] boot success. serializeType=" + RemotingCommand.getSerializeTypeConfigInThisServer();
if (null != controller.getBrokerConfig().getNamesrvAddr()) {
tip += " and name server is " + controller.getBrokerConfig().getNamesrvAddr();
}
log.info(tip);
System.out.printf("%s%n", tip);
return controller;
} catch (Throwable e) {
e.printStackTrace();
System.exit(-1);
}
return null;
}
可以看到，broker中也是有个controller类BrokerController，和nameServer类似。其中createBrokerController(args)方法就是初始化BrokerController的。再createBrokerController(args)主要就做了两件事，一是加载配置文件中的信息到配置对象中；二是根据加载的参数初始化BrokerController对象。下边我们主要看初始化的逻辑。
/**
* 初始化broker的各个模块
* 被{@link BrokerStartup#createBrokerController(String[])}调用
*/
public boolean initialize() throws CloneNotSupportedException {
//加载各种模块
boolean result = this.topicConfigManager.load();
result = result && this.consumerOffsetManager.load();
result = result && this.subscriptionGroupManager.load();
result = result && this.consumerFilterManager.load();
if (result) {
try {
//根据参数构建DefaultMessageStore,此处有开辟对外内存的逻辑
this.messageStore =
new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener,
this.brokerConfig);
if (messageStoreConfig.isEnableDLegerCommitLog()) {
DLedgerRoleChangeHandler roleChangeHandler = new DLedgerRoleChangeHandler(this, (DefaultMessageStore) messageStore);
((DLedgerCommitLog)((DefaultMessageStore) messageStore).getCommitLog()).getdLedgerServer().getdLedgerLeaderElector().addRoleChangeHandler(roleChangeHandler);
}
this.brokerStats = new BrokerStats((DefaultMessageStore) this.messageStore);
//load plugin
MessageStorePluginContext context = new MessageStorePluginContext(messageStoreConfig, brokerStatsManager, messageArrivingListener, brokerConfig);
this.messageStore = MessageStoreFactory.build(context, this.messageStore);
this.messageStore.getDispatcherList().addFirst(new CommitLogDispatcherCalcBitMap(this.brokerConfig, this.consumerFilterManager));
} catch (IOException e) {
result = false;
log.error("Failed to initialize", e);
}
}
/**
* 加载持久化文件：CommitLog、IndexFile
* {@link DefaultMessageStore#load()}
*/
result = result && this.messageStore.load();
if (result) {
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.clientHousekeepingService);
NettyServerConfig fastConfig = (NettyServerConfig) this.nettyServerConfig.clone();
fastConfig.setListenPort(nettyServerConfig.getListenPort() - 2);
this.fastRemotingServer = new NettyRemotingServer(fastConfig, this.clientHousekeepingService);
//初始化各种线程池
this.sendMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getSendMessageThreadPoolNums(),
this.brokerConfig.getSendMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.sendThreadPoolQueue,
new ThreadFactoryImpl("SendMessageThread_"));
this.pullMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getPullMessageThreadPoolNums(),
this.brokerConfig.getPullMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.pullThreadPoolQueue,
new ThreadFactoryImpl("PullMessageThread_"));
this.replyMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getProcessReplyMessageThreadPoolNums(),
this.brokerConfig.getProcessReplyMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.replyThreadPoolQueue,
new ThreadFactoryImpl("ProcessReplyMessageThread_"));
this.queryMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getQueryMessageThreadPoolNums(),
this.brokerConfig.getQueryMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.queryThreadPoolQueue,
new ThreadFactoryImpl("QueryMessageThread_"));
this.adminBrokerExecutor =
Executors.newFixedThreadPool(this.brokerConfig.getAdminBrokerThreadPoolNums(), new ThreadFactoryImpl(
"AdminBrokerThread_"));
this.clientManageExecutor = new ThreadPoolExecutor(
this.brokerConfig.getClientManageThreadPoolNums(),
this.brokerConfig.getClientManageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.clientManagerThreadPoolQueue,
new ThreadFactoryImpl("ClientManageThread_"));
this.heartbeatExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getHeartbeatThreadPoolNums(),
this.brokerConfig.getHeartbeatThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.heartbeatThreadPoolQueue,
new ThreadFactoryImpl("HeartbeatThread_", true));
this.endTransactionExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getEndTransactionThreadPoolNums(),
this.brokerConfig.getEndTransactionThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.endTransactionThreadPoolQueue,
new ThreadFactoryImpl("EndTransactionThread_"));
this.consumerManageExecutor =
Executors.newFixedThreadPool(this.brokerConfig.getConsumerManageThreadPoolNums(), new ThreadFactoryImpl(
"ConsumerManageThread_"));
// 注册各种处理器，用来处理各种网络层的请求
this.registerProcessor();
//启动各种定时任务
final long initialDelay = UtilAll.computeNextMorningTimeMillis() - System.currentTimeMillis();
final long period = 1000 * 60 * 60 * 24;
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.getBrokerStats().record();
} catch (Throwable e) {
log.error("schedule record error.", e);
}
}
}, initialDelay, period, TimeUnit.MILLISECONDS);
// 定时将consumer消费进度持久化
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.consumerOffsetManager.persist();
} catch (Throwable e) {
log.error("schedule persist consumerOffset error.", e);
}
}
}, 1000 * 10, this.brokerConfig.getFlushConsumerOffsetInterval(), TimeUnit.MILLISECONDS);
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.consumerFilterManager.persist();
} catch (Throwable e) {
log.error("schedule persist consumer filter error.", e);
}
}
}, 1000 * 10, 1000 * 10, TimeUnit.MILLISECONDS);
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.protectBroker();
} catch (Throwable e) {
log.error("protectBroker error.", e);
}
}
}, 3, 3, TimeUnit.MINUTES);
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.printWaterMark();
} catch (Throwable e) {
log.error("printWaterMark error.", e);
}
}
}, 10, 1, TimeUnit.SECONDS);
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
log.info("dispatch behind commit log {} bytes", BrokerController.this.getMessageStore().dispatchBehindBytes());
} catch (Throwable e) {
log.error("schedule dispatchBehindBytes error.", e);
}
}
}, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
if (this.brokerConfig.getNamesrvAddr() != null) {
this.brokerOuterAPI.updateNameServerAddressList(this.brokerConfig.getNamesrvAddr());
log.info("Set user specified name server address: {}", this.brokerConfig.getNamesrvAddr());
} else if (this.brokerConfig.isFetchNamesrvAddrByAddressServer()) {
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.brokerOuterAPI.fetchNameServerAddr();
} catch (Throwable e) {
log.error("ScheduledTask fetchNameServerAddr exception", e);
}
}
}, 1000 * 10, 1000 * 60 * 2, TimeUnit.MILLISECONDS);
}
if (!messageStoreConfig.isEnableDLegerCommitLog()) {
if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
if (this.messageStoreConfig.getHaMasterAddress() != null && this.messageStoreConfig.getHaMasterAddress().length() >= 6) {
this.messageStore.updateHaMasterAddress(this.messageStoreConfig.getHaMasterAddress());
this.updateMasterHAServerAddrPeriodically = false;
} else {
this.updateMasterHAServerAddrPeriodically = true;
}
} else {
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.printMasterAndSlaveDiff();
} catch (Throwable e) {
log.error("schedule printMasterAndSlaveDiff error.", e);
}
}
}, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);
}
}
if (TlsSystemConfig.tlsMode != TlsMode.DISABLED) {
// Register a listener to reload SslContext
try {
fileWatchService = new FileWatchService(
new String[] {
TlsSystemConfig.tlsServerCertPath,
TlsSystemConfig.tlsServerKeyPath,
TlsSystemConfig.tlsServerTrustCertPath
},
new FileWatchService.Listener() {
boolean certChanged, keyChanged = false;
@Override
public void onChanged(String path) {
if (path.equals(TlsSystemConfig.tlsServerTrustCertPath)) {
log.info("The trust certificate changed, reload the ssl context");
reloadServerSslContext();
}
if (path.equals(TlsSystemConfig.tlsServerCertPath)) {
certChanged = true;
}
if (path.equals(TlsSystemConfig.tlsServerKeyPath)) {
keyChanged = true;
}
if (certChanged && keyChanged) {
log.info("The certificate and private key changed, reload the ssl context");
certChanged = keyChanged = false;
reloadServerSslContext();
}
}
private void reloadServerSslContext() {
((NettyRemotingServer) remotingServer).loadSslContext();
((NettyRemotingServer) fastRemotingServer).loadSslContext();
}
});
} catch (Exception e) {
log.warn("FileWatchService created error, can't load the certificate dynamically");
}
}
initialTransaction();
initialAcl();
initialRpcHooks();
}
return result;
}
DefaultMessageStore初始化
第一段代码是加载本地文件中的各种信息到系统中，比如topic信息，订阅信息等。紧接着的代码
//根据参数构建DefaultMessageStore,此处有开辟对外内存的逻辑
this.messageStore =
new DefaultMessageStore(this.messageStoreConfig, this.brokerStatsManager, this.messageArrivingListener,
this.brokerConfig);
此段是初始化DefaultMessageStore的。看着是有参构造一个DefaultMessageStore对象，但其中有一段代码
//开启堆外内存并且是异步刷盘
if (messageStoreConfig.isTransientStorePoolEnable()) {
this.transientStorePool.init();
}
this.allocateMappedFileService.start();
条件语句是两个条件，开启堆外内存和异步刷盘，代码this.transientStorePool.init()中是开辟内存的逻辑
/**
* It's a heavy init method.
* 会默认开辟5个（参数transientStorePoolSize控制）堆外内存DirectByteBuffer，循环利用。
* 写消息时，消息都暂存至此，通过线程CommitRealTimeService将数据定时刷到page cache，当数据flush到disk后，再将DirectByteBuffer归还给缓冲池
*/
public void init() {
for (int i = 0; i < poolSize; i++) {
//开辟堆外内存
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(fileSize);
final long address = ((DirectBuffer) byteBuffer).address();
Pointer pointer = new Pointer(address);
LibC.INSTANCE.mlock(pointer, new NativeLong(fileSize));
availableBuffers.offer(byteBuffer);
}
}
CommitLog、ConsumeQueue、IndexFile的初始化
/**
* 加载初始化持久化文件：CommitLog、IndexFile
* {@link DefaultMessageStore#load()}
*/
result = result && this.messageStore.load();
/**
* broker启动时，从磁盘加载文件。用来判断上一次是否是异常退出
* 被BrokerController.initialize()调用
*/
public boolean load() {
boolean result = true;
try {
// 根据是否存在abort文件，判断上一次退出是否正常
boolean lastExitOK = !this.isTempFileExist();
log.info("last shutdown {}", lastExitOK ? "normally" : "abnormally");
if (null != scheduleMessageService) {
result = result && this.scheduleMessageService.load();
}
// load Commit Log
result = result && this.commitLog.load();
// load Consume Queue
result = result && this.loadConsumeQueue();
if (result) {
this.storeCheckpoint =
new StoreCheckpoint(StorePathConfigHelper.getStoreCheckpoint(this.messageStoreConfig.getStorePathRootDir()));
// 加载indexFile
this.indexService.load(lastExitOK);
// 根据是否异常退出，执行不同的恢复策略
this.recover(lastExitOK);
log.info("load over, and the max phy offset = {}", this.getMaxPhyOffset());
}
} catch (Exception e) {
log.error("load exception", e);
result = false;
}
if (!result) {
this.allocateMappedFileService.shutdown();
}
return result;
}
主要逻辑是这几段代码
// load Commit Log
result = result && this.commitLog.load();
// load Consume Queue
result = result && this.loadConsumeQueue();
// 加载indexFile
this.indexService.load(lastExitOK);
这几段代码就是用来构建CommitLog、ConsumeQueue、IndexFile对象。
初始化各种线程池
接下来是初始化各种线程池，比如发送消息的线程池，拉消息的线程池等。例如：
//初始化各种线程池
this.sendMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getSendMessageThreadPoolNums(),
this.brokerConfig.getSendMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.sendThreadPoolQueue,
new ThreadFactoryImpl("SendMessageThread_"));
这是发送消息的线程池，其中核心线程数和最大线程数都是1
/**
* thread numbers for send message thread pool, since spin lock will be used by default since 4.0.x, the default
* value is 1.
*/
private int sendMessageThreadPoolNums = 1; //16 + Runtime.getRuntime().availableProcessors() * 4;
向RemotingServer注册各种处理器
this.remotingServer.registerProcessor(RequestCode.SEND_MESSAGE, sendProcessor, this.sendMessageExecutor);
@Override
public void registerProcessor(int requestCode, NettyRequestProcessor processor, ExecutorService executor) {
ExecutorService executorThis = executor;
if (null == executor) {
executorThis = this.publicExecutor;
}
Pair<NettyRequestProcessor, ExecutorService> pair = new Pair<NettyRequestProcessor, ExecutorService>(processor, executorThis);
this.processorTable.put(requestCode, pair);
}
/**
* This container holds all processors per request code, aka, for each incoming request, we may look up the
* responding processor in this map to handle the request.
*/
protected final HashMap<Integer/* request code */, Pair<NettyRequestProcessor, ExecutorService>> processorTable =
new HashMap<Integer, Pair<NettyRequestProcessor, ExecutorService>>(64);
可以看到这个注册器就是个hashmap，使用时根据code拿到NettyRequestProcessor处理器和对应的线程池。
启动各种定时任务
数据统计，消息进度持久化，数据打印等相关的信息
// 定时将consumer消费进度持久化
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.consumerOffsetManager.persist();
} catch (Throwable e) {
log.error("schedule persist consumerOffset error.", e);
}
}
}, 1000 * 10, this.brokerConfig.getFlushConsumerOffsetInterval(), TimeUnit.MILLISECONDS);
public synchronized void persist() {
String jsonString = this.encode(true);
if (jsonString != null) {
String fileName = this.configFilePath();
try {
MixAll.string2File(jsonString, fileName);
} catch (IOException e) {
log.error("persist file " + fileName + " exception", e);
}
}
}
其中this.encode(true)是调用的ConsumerOffsetManager()类的encode()方法。
其他初始化
initialTransaction();
initialAcl();
initialRpcHooks();
Transaction事务对象的初始化，acl的初始化，rpcHook的初始化。
- rpcHook
RocketMQ的rpcHook是用于拦截和处理远程过程调用（RPC）请求的钩子。它允许用户在消息发送或接收过程中自定义一些额外的逻辑操作。通过实现RocketMQ的RPC Hook接口，您可以在消息发送前或接收后执行一些自定义的逻辑，例如对消息进行加密、解密、验证等操作。rpcHook提供了一种扩展机制，使用户能够灵活地定制RocketMQ的消息传输行为。 - acl
RocketMQ的ACL（Access Control List）是用来控制访问权限的机制。它允许您对RocketMQ中的主题（Topic）和组（Group）进行细粒度的权限管理。通过配置ACL，您可以限制特定用户或角色对特定主题或组的操作权限，包括发送消息、订阅消息、消费消息等。ACL可以帮助您确保只有经过授权的用户或角色能够进行特定的操作，提高系统的安全性和可控性。 - Transaction
RocketMQ的Transaction功能是用于实现分布式事务消息的一种机制。在分布式系统中，事务消息是指需要跨多个参与者（例如消息生产者和消息消费者）的操作保持一致性的消息。通过使用Transaction功能，RocketMQ可以确保在消息发送方和消息接收方之间的事务操作是原子性的，要么全部成功，要么全部失败。
使用Transaction功能时，消息生产者可以将消息发送到RocketMQ Broker，并在本地执行事务操作。然后，消息生产者需要根据事务的执行结果向Broker发送确认消息（Commit或Rollback）。在消息消费者端，可以注册一个事务监听器（TransactionListener），用于处理接收到的事务消息。当事务消息被消费时，消息消费者会根据事务监听器的回调结果来决定是否提交或回滚该消息。
通过Transaction功能，RocketMQ可以提供分布式事务消息的可靠性和一致性，适用于需要跨多个系统进行事务操作的场景，如订单支付、库存扣减等。
总结
- broker也有一个BrokerController，这个是和nameServer一致的。BrokerController封装了很多关于broker的逻辑，比如:broker的初始化，线程池的初始化，通讯组件的初始化，注册各种处理器，初始化各种存储组件等等。
- broker的启动实际就是调用BrokerController的start()逻辑，也就是初始化各种组件。因为底层是依据netty做通信，底层就是启动一个netty做网络通信。
- 启动逻辑中有个注册registerBrokerAll()方法，逻辑底层是遍历所有的nameServer地址，然后注册当前broker到nameServer上。这段逻辑在定时任务内，每个60秒，向nameSrv注册本broker。官网文档中解释：NameServer通常会有多个实例部署，各实例间相互不进行信息通讯。Broker是向每一台NameServer注册自己的路由信息，所以每一个NameServer实例上面都保存一份完整的路由信息。当某个NameServer因某种原因下线了，客户端仍然可以向其它NameServer获取路由信息。
- broker中不同的请求对应着不同的处理器去处理不同的逻辑，初始化的时候会把不同的处理器注册到处理器表中。这个处理器表底层就是个HashMap，key是请求code，value是Pair对象。Pair对象里边包含两个属性，处理器和已初始化的线程池。这样调用方使用的时候就可以根据请求code自动分配处理器和对应的线程池，也对应了代码设计中的单一职责原则和开放-封闭原则。
- 在DefaultMessageStore对象的有参构造函数中，有一个初始化堆外内存的逻辑。this.transientStorePool.init()，这段逻辑只有：开启堆外内存并且消息是异步刷盘到disk 时才会执行。会默认开辟5个（参数transientStorePoolSize控制）堆外内存DirectByteBuffer，循环利用。写消息时，消息都暂存至此，通过线程CommitRealTimeService将数据定时刷到page cache，当数据flush到disk后，再将DirectByteBuffer归还给缓冲池。
上一篇：RocketMQ源码学习三：Broker基础知识之消息存储
下一篇：RocketMQ源码学习五：Producer启动