RocketMQ源码系列（二）RocketMQ路由中心，NameServer源码篇，路由注册，路由发现，路由删除
文章目录
- 1、RocketMQ设计理念
- 2、RocketMQ解决的问题
- 3、NameServer所承担的作用
- 4、NameServer启动流程
- 入口main函数
- 创建Namesrv核心控制器
- start(controller)方法
- controller.initialize()初始化方法
- NamesrvConfig属性
- NettyServerConfig属性
- 5、NameServer路由元信息
- NameServer存储的信息
- 6、NameServer路由注册
- broker发送心跳包
- NameServer处理心跳包
- 维护clusterAddrTable
- 维护brokerAddrTable
- 维护topicQueueTable
- 维护brokerLiveTable
- 维护filterServerTable
- 路由注册总结
- 7、路由删除
- 定时删除路由信息
- scanNotActiveBroker方法
- onChannelDestroy方法逻辑如下
- 8、路由发现
1、RocketMQ设计理念
RocketMQ基于主题的发布与订阅模式，核心包括消息发送、消息存储、消息消费。自研NameServer实现元数据的管理，设计简单，并没有采用Zookeeper作为注册中心。集群之间不保持强一致，追求最终一致性，能容忍分钟级的不一致，所以集群之间互不通信，降低了NameServer的复杂性，网络要求也降低，性能比Zookeeper提高不少。高效的IO存储机制，以及存储分组，组内单个文件大小固定，引入内存映射机制，所有消息基于顺序写，极大地提升写性能。同时引入消息消费队列文件和索引文件，提升了消息消费和查找的效率。
2、RocketMQ解决的问题
顺序消息、消息存储、消息高可用、消息消费低延迟、确保至少被消费一次、回溯消息、消息堆积、定时消息、消息重试等。
3、NameServer所承担的作用
消息中间件一般的设计思路是基于主题的发布订阅机制，消息生产者发送某一主题消息到消息服务器，消息服务器负责消息的持久化存储，消费者订阅感兴趣的主题，消息服务器根据订阅信息将消息推送给消费者，或者消费者主动向消息服务器拉取消息，从而实现生产者与消费者的解耦。那么消息生产者如何知道消息要发往那台消息服务器呢？以及如果有一台消息服务器宕机了，生产者怎么在不重启服务的情况下感知到呢？NameServer就是为了解决这些问题而生。
4、NameServer启动流程
首先打开NameServer启动类NamesrvStartup
入口main函数
启动方法我们可以看到入口函数内容封装成了main0(args)方法，根据参数组装得到NamesrvController对象，然后调用启动方法。
启动方法执行完成后打印日志输出启动成功的信息。
public static void main(String[] args) {
main0(args);
}
public static NamesrvController main0(String[] args) {
try {
//根据参数组装得到NamesrvController对象
NamesrvController controller = createNamesrvController(args);
start(controller);//调用启动方法
String tip = "The Name Server boot success. serializeType=" + RemotingCommand.getSerializeTypeConfigInThisServer();
log.info(tip);
System.out.printf("%s%n", tip);
return controller;
} catch (Throwable e) {
e.printStackTrace();
System.exit(-1);
}
return null;
}
创建Namesrv核心控制器
通过createNamesrvController(args)方法，创建Namesrv核心控制器
public static NamesrvController createNamesrvController(String[] args) throws IOException, JoranException {
//设置key、value键值对的系统属性，添加rocketmq当前版本
System.setProperty(RemotingCommand.REMOTING_VERSION_KEY, Integer.toString(MQVersion.CURRENT_VERSION));
//PackageConflictDetect.detectFastjson();
//构建命令行操作的指令
Options options = ServerUtil.buildCommandlineOptions(new Options());
//得到启动namesrv的命令
commandLine = ServerUtil.parseCmdLine("mqnamesrv", args, buildCommandlineOptions(options), new PosixParser());
if (null == commandLine) {
System.exit(-1);
return null;
}
//创建namesrv业务参数
final NamesrvConfig namesrvConfig = new NamesrvConfig();
//创建namesrv网格参数
final NettyServerConfig nettyServerConfig = new NettyServerConfig();
nettyServerConfig.setListenPort(9876);//设置监听端口
/*
在解析启动时把指定的配置文件或启动命令中的选项值，
填充到namesrvConfig，nettyServerConfig对象中，
参数的来源主要两种方式：
1）-c configFile 通过-c命令指定配置文件的路径
2）使用“--属性名 属性值”的方式入的参，例如--listenPort 9876
*/
if (commandLine.hasOption('c')) {
//如果启动命令指定了配置文件，获取配置文件路径，读取文件内容
String file = commandLine.getOptionValue('c');
if (file != null) {
InputStream in = new BufferedInputStream(new FileInputStream(file));
properties = new Properties();
properties.load(in);
MixAll.properties2Object(properties, namesrvConfig);
MixAll.properties2Object(properties, nettyServerConfig);
//设置命令行启动namesrv指定的配置文件路径
namesrvConfig.setConfigStorePath(file);
System.out.printf("load config properties file OK, %s%n", file);
in.close();
}
}
//控制台打印namesrv的配置信息，命令行上面加p
if (commandLine.hasOption('p')) {
InternalLogger console = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_CONSOLE_NAME);
MixAll.printObjectProperties(console, namesrvConfig);
MixAll.printObjectProperties(console, nettyServerConfig);
System.exit(0);
}
//把命令行属性解析成properties
MixAll.properties2Object(ServerUtil.commandLine2Properties(commandLine), namesrvConfig);
//判断是否设置rocketmqHome。没有设置则报错退出
if (null == namesrvConfig.getRocketmqHome()) {
System.out.printf("Please set the %s variable in your environment to match the location of the RocketMQ installation%n", MixAll.ROCKETMQ_HOME_ENV);
System.exit(-2);
}
//加载rocketmq日志对象
LoggerContext lc = (LoggerContext) LoggerFactory.getILoggerFactory();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(lc);
lc.reset();
/*
读取日志配置文件，此处说明我们的日志配置文件必须是
相对于指定的rocketmqHome下的conf/logback_namesrv.xml
名字固定
*/
configurator.doConfigure(namesrvConfig.getRocketmqHome() + "/conf/logback_namesrv.xml");
log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
MixAll.printObjectProperties(log, namesrvConfig);
MixAll.printObjectProperties(log, nettyServerConfig);
//根据namesrvConfig和nettyServerConfig初始化controller,final不可变更
final NamesrvController controller = new NamesrvController(namesrvConfig, nettyServerConfig);
// remember all configs to prevent discard
//注册配置文件
controller.getConfiguration().registerConfig(properties);
return controller;
}
start(controller)方法
如下，首先对参数进行初始化，然后在jvm中增加一个关闭的钩子，当jvm关闭的时候，会执行系统中已经设置的所有通过方法addShutdownHook添加的钩子，此处当系统执行完controller.shutdown()后，jvm才会关闭。然后调用启动方法。
public static NamesrvController start(final NamesrvController controller) throws Exception {
if (null == controller) {
throw new IllegalArgumentException("NamesrvController is null");
}
boolean initResult = controller.initialize();//对参数进行初始化
if (!initResult) {
//初始化失败则退出
controller.shutdown();
System.exit(-3);
}
/*在jvm中增加一个关闭的钩子，当jvm关闭的时候，
会执行系统中已经设置的所有通过方法addShutdownHook添加的钩子，
当系统执行完controller.shutdown()后，jvm才会关闭。*/
Runtime.getRuntime().addShutdownHook(new ShutdownHookThread(log, new Callable<Void>() {
@Override
public Void call() throws Exception {
controller.shutdown();
return null;
}
}));
controller.start();//调用启动方法
return controller;
}
controller.initialize()初始化方法
public boolean initialize() {
//加载KV配置
this.kvConfigManager.load();
//创建NettyServer网络处理对象
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);
//创建处理网络连接的线程池
this.remotingExecutor =
Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(), new ThreadFactoryImpl("RemotingExecutorThread_"));
//注册业务处理器
this.registerProcessor();
//开启删除不活跃broker的定时任务，间隔10s扫描一次，移除不活跃broker
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.routeInfoManager.scanNotActiveBroker();
}
}, 5, 10, TimeUnit.SECONDS);
//启动一个定时任务，每十分钟打印一次KV配置
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.kvConfigManager.printAllPeriodically();
}
}, 1, 10, TimeUnit.MINUTES);
if (TlsSystemConfig.tlsMode != TlsMode.DISABLED) {
// Register a listener to reload SslContext
// 注册一个监听器去重新加载SslContext
try {
fileWatchService = new FileWatchService(
new String[] {
TlsSystemConfig.tlsServerCertPath,
TlsSystemConfig.tlsServerKeyPath,
TlsSystemConfig.tlsServerTrustCertPath
},
new FileWatchService.Listener() {
boolean certChanged, keyChanged = false;
@Override
public void onChanged(String path) {
if (path.equals(TlsSystemConfig.tlsServerTrustCertPath)) {
log.info("The trust certificate changed, reload the ssl context");
reloadServerSslContext();
}
if (path.equals(TlsSystemConfig.tlsServerCertPath)) {
certChanged = true;
}
if (path.equals(TlsSystemConfig.tlsServerKeyPath)) {
keyChanged = true;
}
if (certChanged && keyChanged) {
log.info("The certificate and private key changed, reload the ssl context");
certChanged = keyChanged = false;
reloadServerSslContext();
}
}
private void reloadServerSslContext() {
((NettyRemotingServer) remotingServer).loadSslContext();
}
});
} catch (Exception e) {
log.warn("FileWatchService created error, can't load the certificate dynamically");
}
}
return true;
}
NamesrvConfig属性
public class NamesrvConfig {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
/*
* rocketmqHome表示主目录，可以通过如下两种方式配置RocketMQ的主目录
* 1）-Drocketmq.home.dir=path
* 2）通过设置环境变量ROCKETMQ_HOME
*/
private String rocketmqHome = System.getProperty(MixAll.ROCKETMQ_HOME_PROPERTY, System.getenv(MixAll.ROCKETMQ_HOME_ENV));
//NameServer存储KV配置属性的持久化路径。
private String kvConfigPath = System.getProperty("user.home") + File.separator + "namesrv" + File.separator + "kvConfig.json";
/*
* 默认配置文件路径，但是不生效。nameServer启动时，
* 如果要通过配置文件配置NameServer启动属性的话，
* 要使用-c选项指定配置文件。
*/
private String configStorePath = System.getProperty("user.home") + File.separator + "namesrv" + File.separator + "namesrv.properties";
//集群测试使用的参数
private String productEnvName = "center";
//是否是集群测试，默认为false
private boolean clusterTest = false;
//是否支持顺序消息，默认是不支持。
private boolean orderMessageEnable = false;
NettyServerConfig属性
public class NettyServerConfig implements Cloneable {
//nameserver监听端口，该值会默认被初始化为9876
private int listenPort = 8888;
//Netty业务线程池线程个数
private int serverWorkerThreads = 8;
/*
Netty public任务线程池线程个数，Netty网络设计会根据业务类型创建不同的线程池
如消息发送、消息消费、心跳检测等业务场景，如果某个业务类型未注册到线程池，
则由public线程池执行。
*/
private int serverCallbackExecutorThreads = 0;
/*
IO线程池个数，主要是NameServer、Broker端解析请求、返回相应的线程个数，这类线程
主要是处理网络请求的，解析请求包，然后转发到各个业务线程池完成具体业务操作，
再将结果返回给调用方。
*/
private int serverSelectorThreads = 3;
//send oneway类型的消息请求并发度（Broker端参数）
private int serverOnewaySemaphoreValue = 256;
//异步消息请求最大并发度（Broker端参数）
private int serverAsyncSemaphoreValue = 64;
/*
网络连接最大空闲时间，默认120s，如果连接空闲时间超过该参数设置的值，连接将被关闭。
*/
private int serverChannelMaxIdleTimeSeconds = 120;
//网络socket发送缓存区大小，默认是64k
private int serverSocketSndBufSize = NettySystemConfig.socketSndbufSize;
//网络socket接收缓存区大小，默认是64k
private int serverSocketRcvBufSize = NettySystemConfig.socketRcvbufSize;
//ByteBuffer是否开启缓存，建议是开启
private boolean serverPooledByteBufAllocatorEnable = true;
/**
* make make install
* 是否启用Epll IO模型，Linux环境建议开启
*
* ../glibc-2.10.1/configure \ --prefix=/usr \ --with-headers=/usr/include \
* --host=x86_64-linux-gnu \ --build=x86_64-pc-linux-gnu \ --without-gd
*/
private boolean useEpollNativeSelector = false;
5、NameServer路由元信息
NameServer主要是为了给生产者和消费者提供topic的路由信息，那么NameServer需要存储那些信息呢？并且还要管理Broker节点，包括路由注册，路由删除等。
NameServer存储的信息
首先找到RouteInfoManager
代码解释如下
public class RouteInfoManager {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
private final static long BROKER_CHANNEL_EXPIRED_TIME = 1000 * 60 * 2;
private final ReadWriteLock lock = new ReentrantReadWriteLock();
//Topic消息队列路由信息，消息发送时根据路由表进行负载均衡
private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
//存储broker的基本信息，比如名称、所属集群、主备地址等
private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
//存储broker集群中所有broker名称，
private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
//broker状态表，每次nameserver收到心跳包时更新此表信息
private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
//broker上的filterServer列表，用于类模式消息过滤
private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
RocketMQ中一个Topic拥有多个消息队列，默认情况下一个broker为每一个topic创建4个读队列4个写队列。多个broker组成一个集群，brokerName由相同的多台broker组成Master-Slave架构，brokerId为0代表master，大于0表示Slave。BrokerLiveInfo中的lastUpdateTimestamp属性存储上次收到心跳包的时间。
6、NameServer路由注册
RocketMQ路由注册时通过broker与nameserver的心跳机制实现的。broker启动时向所有集群中的nameserver发送心跳请求，每隔30s向所有集群中的nameserver发送心跳包，nameserver收到broker心跳包时会更新brokerLiveTable缓存中brokerLiveInfo的lastUpdateTimestamp字段，然后nameserver每隔10s扫描brokerLiveTable，如果连续120s没有收到心跳包，nameserver将移除该broker的路由信息同时关闭socket连接。
broker发送心跳包
BrokerController的start方法中添加一个定时任务，执行BrokerController的registerBrokerAll方法，向所有的nameserver注册路由信息。
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
} catch (Throwable e) {
log.error("registerBrokerAll Exception", e);
}
}
}, 1000 * 10, Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS);
BrokerController的registerBrokerAll方法调用doRegisterBrokerAll方法，doRegisterBrokerAll调用BrokerOuterAPI的registerBrokerAll方法进行注册
start
——>registerBrokerAll(BrokerController)
——>doRegisterBrokerAll(BrokerController)
——>registerBrokerAll(BrokerOuterAPI)
registerBrokerAll(BrokerOuterAPI)方法遍历nameserver列表，依次向nameserver发送心跳包。
final CountDownLatch countDownLatch = new CountDownLatch(nameServerAddressList.size());
//遍历nameserver列表，依次向nameserver发送心跳包。
for (final String namesrvAddr : nameServerAddressList) {
brokerOuterExecutor.execute(new Runnable() {
@Override
public void run() {
try {
RegisterBrokerResult result = registerBroker(namesrvAddr,oneway, timeoutMills,requestHeader,body);
if (result != null) {
registerBrokerResultList.add(result);
}
log.info("register broker[{}]to name server {} OK", brokerId, namesrvAddr);
} catch (Exception e) {
log.warn("registerBroker Exception, {}", namesrvAddr, e);
} finally {
countDownLatch.countDown();
}
}
});
}
try {
countDownLatch.await(timeoutMills, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
}
最终调用registerBroker(BrokerOuterAPI)发送心跳包，底层传输基于Netty。
NameServer处理心跳包
RouteInfoManager的registerBroker处理心跳包
维护clusterAddrTable
首先加锁写防止并发修改路由表，判断该集群是否存在，获取该集群对应的broker，如果不存在则创建集群，将broker加入集群中
//加锁写防止并发修改路由表。
this.lock.writeLock().lockInterruptibly();
//判断该集群是否存在，获取该集群对应的broker
Set<String> brokerNames = this.clusterAddrTable.get(clusterName);
if (null == brokerNames) {
brokerNames = new HashSet<String>();
//如果不存在则创建集群
this.clusterAddrTable.put(clusterName, brokerNames);
}
//将broker加入集群中
brokerNames.add(brokerName);
维护brokerAddrTable
首先从brokerAddrTable根据BrokerName尝试获取Broker信息，如果不存在，则新建BrokerData对象并放入到brokerAddrTable中，把registerFirst设置为true；如果存在，直接替换原先的，registerFirst为false。
//是否是第一次注册，false表示不是首次注册
boolean registerFirst = false;
//尝试从brokerAddrTable中获取BrokerData信息
BrokerData brokerData = this.brokerAddrTable.get(brokerName);
if (null == brokerData) {
//如果为空则表示首次注册registerFirst置为true
registerFirst = true;
//新建brokerdata对象
brokerData = new BrokerData(clusterName, brokerName, new HashMap<Long, String>());
//添加到brokerAddrTable中
this.brokerAddrTable.put(brokerName, brokerData);
}
维护topicQueueTable
如果broker为master，并且topic信息发生变化或者是首次注册，则需要更新或者创建topic路由元数据，填入topicQueueTable，实际上就是为默认主题自动注册路由信息，其中包含MixAll.DEFAULT_TOPIC的路由信息。当生产者发送topic时，如果该topic未创建并且BrokerConfig的autoCreateTopicEnable为true时，将返回MixAll.DEFAULT_TOPIC的路由信息。
if (null != topicConfigWrapper
&& MixAll.MASTER_ID == brokerId) {
if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
|| registerFirst) {
ConcurrentMap<String, TopicConfig> tcTable =
topicConfigWrapper.getTopicConfigTable();
if (tcTable != null) {
for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
//维护topicQueueTable
this.createAndUpdateQueueData(brokerName, entry.getValue());
}
}
}
}
private void createAndUpdateQueueData(final String brokerName, final TopicConfig topicConfig) {
QueueData queueData = new QueueData();
queueData.setBrokerName(brokerName);
queueData.setWriteQueueNums(topicConfig.getWriteQueueNums());
queueData.setReadQueueNums(topicConfig.getReadQueueNums());
queueData.setPerm(topicConfig.getPerm());//设置队列读写权限
queueData.setTopicSynFlag(topicConfig.getTopicSysFlag());//topic同步标记
//判断topicQueueTable中是否存在topic对应的队列信息
List<QueueData> queueDataList = this.topicQueueTable.get(topicConfig.getTopicName());
if (null == queueDataList) {
queueDataList = new LinkedList<QueueData>();
//不存在则创建topic路由信息
queueDataList.add(queueData);
//并添加到topicQueueTable
this.topicQueueTable.put(topicConfig.getTopicName(), queueDataList);
log.info("new topic registered, {} {}", topicConfig.getTopicName(), queueData);
} else {
//如果存在则和最新的路由信息进行比对
boolean addNewOne = true;
//遍历topicQueueTable中的路由信息
Iterator<QueueData> it = queueDataList.iterator();
while (it.hasNext()) {
QueueData qd = it.next();
if (qd.getBrokerName().equals(brokerName)) {
//当brokerName相同，对应的queueData也相同时，则说明没有新加broker信息
if (qd.equals(queueData)) {
addNewOne = false;
} else {
//当brokerName相同，对应的queueData不同时，
// 说明broker信息发生变化，移除旧的信息
log.info("topic changed, {} OLD: {} NEW: {}", topicConfig.getTopicName(), qd,
queueData);
it.remove();
}
}
}
//broker信息发生变化时，移除旧的信息后，添加新的信息
if (addNewOne) {
queueDataList.add(queueData);
}
}
}
维护brokerLiveTable
更新brokerLiveTable存活信息表，brokerLiveTable是执行路由删除的重要依据。
//添加新的信息返回旧的信息
BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
new BrokerLiveInfo(
System.currentTimeMillis(),
topicConfigWrapper.getDataVersion(),
channel,
haServerAddr));
if (null == prevBrokerLiveInfo) {
log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
}
维护filterServerTable
一个broker会关联多个filterServer消息过滤器。
if (filterServerList != null) {
if (filterServerList.isEmpty()) {
this.filterServerTable.remove(brokerAddr);
} else {
this.filterServerTable.put(brokerAddr, filterServerList);
}
}
//如果此broker不是master，则需要查找该broker对应的master，
// 并更新对应的masterAddr
if (MixAll.MASTER_ID != brokerId) {
String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
if (masterAddr != null) {
BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
if (brokerLiveInfo != null) {
result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
result.setMasterAddr(masterAddr);
}
}
}
路由注册总结
broker与nameserver进行长连接，broker状态存储在brokerLiveTable中，nameserver每收到一个心跳包，将更新brokerLiveTable中关于broker的状态信息以及路由表（上面更新的几个table），更新时使用了锁粒度较少的读写锁，允许多个生产者并发读取，保证消息发送时的高并发。但同一时刻nameserver只处理一个broker心跳包，多个心跳包处理串行执行。这也是读写锁经典用法。
7、路由删除
当broker宕机，nameserver无法收到心跳包，此时nameserver如何来剔除这些失效的broker呢？nameserver会每隔10s扫描brokerLiveTable状态表，如果BrokerLive的lastUpdateTimestamp的时间戳距离当前时间超过120s，则认为broker失效，移除该broker，关闭与broker连接，并同时更新记录的一系列table。
rocketmq有两种方式来出发路由删除：
1）nameserver定时扫描brokerLiveTable,如果BrokerLive的lastUpdateTimestamp的时间戳距离当前时间超过120s，则认为broker失效，移除该broker.
2)broker正常关闭时，执行unregisterBroker指令。主动触发路由删除。
两种方式删除的逻辑都是一样的。
定时删除路由信息
前面我们知道在nameserver启动时，启动了两个定时任务，其中一个就是每隔10s定时扫描brokerLiveTable，移除不活跃broker。
public boolean initialize() {
//加载KV配置
this.kvConfigManager.load();
//创建NettyServer网络处理对象
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);
//创建处理网络连接的线程池
this.remotingExecutor =
Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(), new ThreadFactoryImpl("RemotingExecutorThread_"));
//注册业务处理器
this.registerProcessor();
//开启删除不活跃broker的定时任务，间隔10s扫描一次，移除不活跃broker
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.routeInfoManager.scanNotActiveBroker();
}
}, 5, 10, TimeUnit.SECONDS);
//启动一个定时任务，每十分钟打印一次KV配置
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.kvConfigManager.printAllPeriodically();
}
}, 1, 10, TimeUnit.MINUTES);
scanNotActiveBroker方法
由上面看出，调用routeInfoManager.scanNotActiveBroker()方法进行路由删除
public void scanNotActiveBroker() {
//遍历brokerLiveTable路由表
Iterator<Entry<String, BrokerLiveInfo>> it = this.brokerLiveTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, BrokerLiveInfo> next = it.next();
//获取lastUpdateTimestamp
long last = next.getValue().getLastUpdateTimestamp();
//判断如果lastUpdateTimestamp的时间加上120，并且小于当前系统时间
if ((last + BROKER_CHANNEL_EXPIRED_TIME) < System.currentTimeMillis()) {
//关闭Channel
RemotingUtil.closeChannel(next.getValue().getChannel());
//移除此broker
it.remove();
log.warn("The broker channel expired, {} {}ms", next.getKey(), BROKER_CHANNEL_EXPIRED_TIME);
//删除与该broker相关的路由信息
this.onChannelDestroy(next.getKey(), next.getValue().getChannel());
}
}
}
onChannelDestroy方法逻辑如下
public void onChannelDestroy(String remoteAddr, Channel channel) {
String brokerAddrFound = null;
if (channel != null) {
try {
try {
//获取读锁
this.lock.readLock().lockInterruptibly();
Iterator<Entry<String, BrokerLiveInfo>> itBrokerLiveTable =
this.brokerLiveTable.entrySet().iterator();
while (itBrokerLiveTable.hasNext()) {
Entry<String, BrokerLiveInfo> entry = itBrokerLiveTable.next();
//根据channel，找到对应的brokerAddr
if (entry.getValue().getChannel() == channel) {
brokerAddrFound = entry.getKey();
break;
}
}
} finally {
//释放读锁
this.lock.readLock().unlock();
}
} catch (Exception e) {
log.error("onChannelDestroy Exception", e);
}
}
if (null == brokerAddrFound) {
brokerAddrFound = remoteAddr;
} else {
log.info("the broker's channel destroyed, {}, clean it's data structure at once", brokerAddrFound);
}
if (brokerAddrFound != null && brokerAddrFound.length() > 0) {
try {
try {
//申请写锁，根据brokerAddrFound从brokerLiveTable、filterServerTable中移除
this.lock.writeLock().lockInterruptibly();
this.brokerLiveTable.remove(brokerAddrFound);
this.filterServerTable.remove(brokerAddrFound);
String brokerNameFound = null;
boolean removeBrokerName = false;
//遍历brokerAddrTable
Iterator<Entry<String, BrokerData>> itBrokerAddrTable =
this.brokerAddrTable.entrySet().iterator();
while (itBrokerAddrTable.hasNext() && (null == brokerNameFound)) {
//获取BrokerData
BrokerData brokerData = itBrokerAddrTable.next().getValue();
//从BrokerData中获取BrokerAddrs
Iterator<Entry<Long, String>> it = brokerData.getBrokerAddrs().entrySet().iterator();
while (it.hasNext()) {
//遍历BrokerAddrs
Entry<Long, String> entry = it.next();
Long brokerId = entry.getKey();
//拿到每一个brokerAddr
String brokerAddr = entry.getValue();
if (brokerAddr.equals(brokerAddrFound)) {
//找到具体的broker，从BrokerData中移除
brokerNameFound = brokerData.getBrokerName();
it.remove();
log.info("remove brokerAddr[{}, {}] from brokerAddrTable, because channel destroyed",
brokerId, brokerAddr);
break;
}
}
//如果移除后，在brokerData中不再包含其他broker，则在brokerAddressTable中
//移除该brokerName对应的条目
if (brokerData.getBrokerAddrs().isEmpty()) {
removeBrokerName = true;
itBrokerAddrTable.remove();
log.info("remove brokerName[{}] from brokerAddrTable, because channel destroyed",
brokerData.getBrokerName());
}
}
if (brokerNameFound != null && removeBrokerName) {
//从clusterAddrTable中找到并从集群中移除。
Iterator<Entry<String, Set<String>>> it = this.clusterAddrTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, Set<String>> entry = it.next();
String clusterName = entry.getKey();
Set<String> brokerNames = entry.getValue();
boolean removed = brokerNames.remove(brokerNameFound);
if (removed) {
log.info("remove brokerName[{}], clusterName[{}] from clusterAddrTable, because channel destroyed",
brokerNameFound, clusterName);
//移除后，集群中不包含任何broker，则将该集群从clusterAddrTable中移除
if (brokerNames.isEmpty()) {
log.info("remove the clusterName[{}] from clusterAddrTable, because channel destroyed and no broker in this cluster",
clusterName);
it.remove();
}
break;
}
}
}
if (removeBrokerName) {
//遍历所有主题的队列
Iterator<Entry<String, List<QueueData>>> itTopicQueueTable =
this.topicQueueTable.entrySet().iterator();
while (itTopicQueueTable.hasNext()) {
Entry<String, List<QueueData>> entry = itTopicQueueTable.next();
String topic = entry.getKey();
List<QueueData> queueDataList = entry.getValue();
Iterator<QueueData> itQueueData = queueDataList.iterator();
while (itQueueData.hasNext()) {
QueueData queueData = itQueueData.next();
//如果队列中包含了当前broker的队列，则移除
if (queueData.getBrokerName().equals(brokerNameFound)) {
itQueueData.remove();
log.info("remove topic[{} {}], from topicQueueTable, because channel destroyed",
topic, queueData);
}
}
//如果topic只包含待移除broker的队列的话，从路由表中删除该topic
if (queueDataList.isEmpty()) {
itTopicQueueTable.remove();
log.info("remove topic[{}] all queue, from topicQueueTable, because channel destroyed",
topic);
}
}
}
} finally {
//释放锁，完成路由删除
this.lock.writeLock().unlock();
}
} catch (Exception e) {
log.error("onChannelDestroy Exception", e);
}
}
}
8、路由发现
当topic路由发生变化，nameserver不会主动统治客户端，而是由客户端定时拉取主题最新的路由信息。
DefaultRequestProcessor类中根据topic获取路由信息的方法getRouteInfoByTopic
public RemotingCommand getRouteInfoByTopic(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(null);
final GetRouteInfoRequestHeader requestHeader =
(GetRouteInfoRequestHeader) request.decodeCommandCustomHeader(GetRouteInfoRequestHeader.class);
/*
调用RouteInfoManager方法，从路由表topicQueueTable、brokerAddrTable、
filterServerTable中分别填充TopicRouteData中的List、List
和filterServerTable地址表
*/
TopicRouteData topicRouteData = this.namesrvController.getRouteInfoManager().pickupTopicRouteData(requestHeader.getTopic());
if (topicRouteData != null) {
//获取到对应的路由信息
//如果该主题为顺序消息
if (this.namesrvController.getNamesrvConfig().isOrderMessageEnable()) {
//从NameServerKVconfig中获取关于顺序消息相关的配置
String orderTopicConf =
this.namesrvController.getKvConfigManager().getKVConfig(NamesrvUtil.NAMESPACE_ORDER_TOPIC_CONFIG,
requestHeader.getTopic());
//填充路由信息
topicRouteData.setOrderTopicConf(orderTopicConf);
}
byte[] content = topicRouteData.encode();
response.setBody(content);
response.setCode(ResponseCode.SUCCESS);
response.setRemark(null);
return response;
}
//如果没找到则使用TOPIC_NOT_EXIST，表示没有找到对应的路由
response.setCode(ResponseCode.TOPIC_NOT_EXIST);
response.setRemark("No topic route info in name server for the topic: " + requestHeader.getTopic()
+ FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL));
return response;
}