RocketMQ原理源码分析之路由注册
RocketMQ原理源码分析之路由注册
- 参考文献
- RocketMQ整体架构
- NameServer
- Broker
- 总结
参考文献
《RocketMQ技术内幕》
RocketMQ整体架构
RocketMQ的主要角色有，轻量级注册中心NameServer、消息存储即流程控制器Broker、消息生产者Producer、消息消费者Consumer。整体架构图如下
整体的工作流程是Broker在启动时会向所有NameServer发起注册，NameServer本地利用二级缓存对Broker的路由信息进行存储，两者之间利用心跳机制对注册的信息进行维护。Producer在发送消息时，会先向NameServer获取Broker的路由信息再进行消息的发送。Consumer在消费时同样会先向NameServer获取Broker的路由信息后找到对应的消费队列进行消息的消费，这里的消费方式有推、拉两种模式。
所以说上图有一个地方画错了，下面源码分析的时候揭晓。
NameServer
org.apache.rocketmq.namesrv.NamesrvStartup#createNamesrvController
System.setProperty(RemotingCommand.REMOTING_VERSION_KEY, Integer.toString(MQVersion.CURRENT_VERSION));
Options options = ServerUtil.buildCommandlineOptions(new Options());
commandLine = ServerUtil.parseCmdLine("mqnamesrv", args, buildCommandlineOptions(options), new PosixParser());
if (null == commandLine) {
System.exit(-1);
return null;
}
// NameServer的基本配置
final NamesrvConfig namesrvConfig = new NamesrvConfig();
// NameServer的网络相关的配置
final NettyServerConfig nettyServerConfig = new NettyServerConfig();
// 默认端口
nettyServerConfig.setListenPort(9876);
// 根据启动命令行来读取配置文件加载配置
if (commandLine.hasOption('c')) {
String file = commandLine.getOptionValue('c');
if (file != null) {
InputStream in = new BufferedInputStream(new FileInputStream(file));
properties = new Properties();
properties.load(in);
MixAll.properties2Object(properties, namesrvConfig);
MixAll.properties2Object(properties, nettyServerConfig);
namesrvConfig.setConfigStorePath(file);
System.out.printf("load config properties file OK, %s%n", file);
in.close();
}
}
// 根据启动命令行的指令加载配置
if (commandLine.hasOption('p')) {
InternalLogger console = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_CONSOLE_NAME);
MixAll.printObjectProperties(console, namesrvConfig);
MixAll.printObjectProperties(console, nettyServerConfig);
System.exit(0);
}
org.apache.rocketmq.namesrv.NamesrvController#initialize
// 该方法注册了NameServer处理netty请求的统一处理类，也是RocketMQ实现各组件通信的关键
// 后面会细说
this.registerProcessor();
// 每10秒进行一次broker的心跳检测，如果超过120秒没有收到broker的心跳包
// 则移除broker的注册信息，具体的实现在scanNotActiveBroker()方法中
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.routeInfoManager.scanNotActiveBroker();
}
}, 5, 10, TimeUnit.SECONDS);
// 每10秒打印一次NameServer的配置信息
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.kvConfigManager.printAllPeriodically();
}
}, 1, 10, TimeUnit.MINUTES);
org.apache.rocketmq.namesrv.NamesrvStartup#start
// 由于NameServer在启动时用到了固定周期的线程池，为了保证NameServer在关闭时优雅停机
// 这里注册了一个JVM的钩子，保证NameServer在关闭时对所有的线程池进行shotdown操作，防止任务的丢失
Runtime.getRuntime().addShutdownHook(new ShutdownHookThread(log, new Callable<Void>() {
@Override
public Void call() throws Exception {
controller.shutdown();
return null;
}
}));
RocketMQ在对注册中心进行选型时，没有使用常用的Zookeeper、NameServer等，而选择自研更加轻量级的注册中心NameServer。原因如下，集群中的每个NameServer是独立的，无状态的各节点省去了主从选举、消息同步等流程，从而保证了极致的性能，这也是为什么Broker在启动时需要向所有NameServer进行注册的原因。
Broker
org.apache.rocketmq.broker.BrokerController#initialize
if (this.brokerConfig.getNamesrvAddr() != null) {
// 把broker配置中的nameServer地址更新到org.apache.rocketmq.remoting.netty.NettyRemotingClient#namesrvAddrList
this.brokerOuterAPI.updateNameServerAddressList(this.brokerConfig.getNamesrvAddr());
log.info("Set user specified name server address: {}", this.brokerConfig.getNamesrvAddr());
} else if (this.brokerConfig.isFetchNamesrvAddrByAddressServer()) {
// fetchNamesrvAddrByAddressServer默认值为false，如果配置为true
// 则每两分钟获取重新拉取并维护NameServer的路由信息
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.brokerOuterAPI.fetchNameServerAddr();
} catch (Throwable e) {
log.error("ScheduledTask fetchNameServerAddr exception", e);
}
}
}, 1000 * 10, 1000 * 60 * 2, TimeUnit.MILLISECONDS);
}
org.apache.rocketmq.broker.BrokerController#start
// 根据配置的时间间隔，最短10秒、最长1分钟向所有的NameServer发起路由注册
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
} catch (Throwable e) {
log.error("registerBrokerAll Exception", e);
}
}
}, 1000 * 10, Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS);
org.apache.rocketmq.broker.out.BrokerOuterAPI#registerBrokerAll
final List<RegisterBrokerResult> registerBrokerResultList = new CopyOnWriteArrayList<>();
// 这个就是在BrokerController的initialize方法中在RemotingClient中维护NameServer地址
List<String> nameServerAddressList = this.remotingClient.getNameServerAddressList();
if (nameServerAddressList != null && nameServerAddressList.size() > 0) {
// 构建注册路由信息的请求
final RegisterBrokerRequestHeader requestHeader = new RegisterBrokerRequestHeader();
// broker地址
requestHeader.setBrokerAddr(brokerAddr);
// 是否主节点 0-主 大于等于1 从
requestHeader.setBrokerId(brokerId);
// broker名称
requestHeader.setBrokerName(brokerName);
// 集群名称
requestHeader.setClusterName(clusterName);
// 高可用地址
requestHeader.setHaServerAddr(haServerAddr);
// 是否进行请求体的压缩
requestHeader.setCompressed(compressed);
RegisterBrokerBody requestBody = new RegisterBrokerBody();
requestBody.setTopicConfigSerializeWrapper(topicConfigWrapper);
requestBody.setFilterServerList(filterServerList);
final byte[] body = requestBody.encode(compressed);
final int bodyCrc32 = UtilAll.crc32(body);
requestHeader.setBodyCrc32(bodyCrc32);
final CountDownLatch countDownLatch = new CountDownLatch(nameServerAddressList.size());
for (final String namesrvAddr : nameServerAddressList) {
brokerOuterExecutor.execute(new Runnable() {
@Override
public void run() {
try {
// 多线程同时注册，保证最高的效率
RegisterBrokerResult result = registerBroker(namesrvAddr,oneway, timeoutMills,requestHeader,body);
if (result != null) {
registerBrokerResultList.add(result);
}
log.info("register broker[{}]to name server {} OK", brokerId, namesrvAddr);
} catch (Exception e) {
log.warn("registerBroker Exception, {}", namesrvAddr, e);
} finally {
countDownLatch.countDown();
}
}
});
}
try {
// 利用线程计数来保证上面的多线程注册都完成
countDownLatch.await(timeoutMills, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
}
org.apache.rocketmq.broker.out.BrokerOuterAPI#registerBroker
// RocketMQ各个组件之前都是基于netty通信的，底层原理这里不细说，只说怎么定位请求的方法
// RequestCode.REGISTER_BROKER为本次请求的标识，全局搜索便可以在DefaultRequestProcessor类找到对应的处理逻辑
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.REGISTER_BROKER, requestHeader);
request.setBody(body);
if (oneway) {
try {
this.remotingClient.invokeOneway(namesrvAddr, request, timeoutMills);
} catch (RemotingTooMuchRequestException e) {
// Ignore
}
return null;
}
RemotingCommand response = this.remotingClient.invokeSync(namesrvAddr, request, timeoutMills);
assert response != null;
switch (response.getCode()) {
case ResponseCode.SUCCESS: {
RegisterBrokerResponseHeader responseHeader =
(RegisterBrokerResponseHeader) response.decodeCommandCustomHeader(RegisterBrokerResponseHeader.class);
RegisterBrokerResult result = new RegisterBrokerResult();
result.setMasterAddr(responseHeader.getMasterAddr());
result.setHaServerAddr(responseHeader.getHaServerAddr());
if (response.getBody() != null) {
result.setKvTable(KVTable.decode(response.getBody(), KVTable.class));
}
return result;
}
default:
break;
}
org.apache.rocketmq.namesrv.NamesrvController#registerProcessor
// NameServer在初始化的时候调用该方法，将DefaultRequestProcessor注册为netty请求的统一处理类
// 所以当NameServer接收到请求时，DefaultRequestProcessor会根据请求头中的code标识进行不同的处理
if (namesrvConfig.isClusterTest()) {
this.remotingServer.registerDefaultProcessor(new ClusterTestRequestProcessor(this, namesrvConfig.getProductEnvName()),
this.remotingExecutor);
} else {
this.remotingServer.registerDefaultProcessor(new DefaultRequestProcessor(this), this.remotingExecutor);
}
org.apache.rocketmq.namesrv.routeinfo.RouteInfoManager#registerBroker
// DefaultRequestProcessor根据请求头带有RequestCode.REGISTER_BROKER会调用该方法
RegisterBrokerResult result = new RegisterBrokerResult();
try {
try {
// ReadWriteLock 这里是串行写
this.lock.writeLock().lockInterruptibly();
Set<String> brokerNames = this.clusterAddrTable.get(clusterName);
if (null == brokerNames) {
brokerNames = new HashSet<String>();
// 维护集群名称和broker名称的映射关系
this.clusterAddrTable.put(clusterName, brokerNames);
}
brokerNames.add(brokerName);
boolean registerFirst = false;
BrokerData brokerData = this.brokerAddrTable.get(brokerName);
if (null == brokerData) {
registerFirst = true;
brokerData = new BrokerData(clusterName, brokerName, new HashMap<Long, String>());
// 维护broker名称和broker具体路由信息的映射关系
this.brokerAddrTable.put(brokerName, brokerData);
}
Map<Long, String> brokerAddrsMap = brokerData.getBrokerAddrs();
// 这里的作用主要是检测时候是否有broker的主从状态发生了变化
// 保证brokerAddrsMap对应的broker信息只有一份
Iterator<Entry<Long, String>> it = brokerAddrsMap.entrySet().iterator();
while (it.hasNext()) {
Entry<Long, String> item = it.next();
if (null != brokerAddr && brokerAddr.equals(item.getValue()) && brokerId != item.getKey()) {
it.remove();
}
}
String oldAddr = brokerData.getBrokerAddrs().put(brokerId, brokerAddr);
registerFirst = registerFirst || (null == oldAddr);
if (null != topicConfigWrapper
&& MixAll.MASTER_ID == brokerId) {
if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
|| registerFirst) {
ConcurrentMap<String, TopicConfig> tcTable =
topicConfigWrapper.getTopicConfigTable();
if (tcTable != null) {
for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
this.createAndUpdateQueueData(brokerName, entry.getValue());
}
}
}
}
// 维护broker的心跳信息，用于NameServer的心跳检测
// NameServerc初始化时启动的心跳检测线程池就是从brokerLiveTable中进行检测
BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
new BrokerLiveInfo(
System.currentTimeMillis(),
topicConfigWrapper.getDataVersion(),
channel,
haServerAddr));
if (null == prevBrokerLiveInfo) {
log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
}
if (filterServerList != null) {
if (filterServerList.isEmpty()) {
this.filterServerTable.remove(brokerAddr);
} else {
this.filterServerTable.put(brokerAddr, filterServerList);
}
}
if (MixAll.MASTER_ID != brokerId) {
String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
if (masterAddr != null) {
BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
if (brokerLiveInfo != null) {
result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
result.setMasterAddr(masterAddr);
}
}
}
} finally {
this.lock.writeLock().unlock();
}
} catch (Exception e) {
log.error("registerBroker Exception", e);
}
return result;
总结
先回到文章开头的问题。由于Broker是向所有的NameServer进行注册，细心的读者已经发现BrokerA-Master少向一个NameServer注册了。下面是路由注册的工作流程图。
最后说一下看完路由注册模块源码的一些想法。技术服务于实际的运用场景，RocketMQ最核心的设计理念我认为就是快，一切的设计都是为了保证最高的性能和效率，这点在NameServer上体现得淋漓尽致。Eureka主从信息同步、心跳检测耗费性能，Zookeeper主从选举期间导致的不可用，在NameServer的设计理念中通通都被砍掉。当然，这也会引起其他的问题，比如Broker的宕机，Producer最长需要130秒才能感知到，这期间的消息发送失败问题需要解决。路由注册模块运用了大量了本地缓存用来提升性能，并且每次Broker与NameServer的心跳包中，都是包含了全量的路由信息，确保了NameServer发生宕机或者网络分区的情况下，能快速的恢复路由信息，最大的保证了高可用。
最后的最后说点妙的东西，很多的源码中对状态位的设置都是2的次方，而在日常工作中我们对于状态位的数据字典更多是0123。我们来看看RocketMQ中对状态位的处理是什么样的
org.apache.rocketmq.common.constant.PermName
public static final int PERM_READ = 0x1 << 2; // 4
public static final int PERM_WRITE = 0x1 << 1;// 2
public static final int PERM_INHERIT = 0x1 << 0; // 1
public static boolean isReadable(final int perm) {
return (perm & PERM_READ) == PERM_READ;
}
public static boolean isWriteable(final int perm) {
return (perm & PERM_WRITE) == PERM_WRITE;
}
public static boolean isInherited(final int perm) {
return (perm & PERM_INHERIT) == PERM_INHERIT;
}
这里对于是否同时具有读写权限的操作可以通过位运算轻松得出，如果使用0123这种状态位的话，则需要写(perm == read || perm == write)。当然，这种写方法看起来更简单名了，但是位运算的性能是最高的，作为一个有追求的程序员，使用二进制的状态位会更能体现出水平。