RocketMQ之Namesrv篇：优雅的服务管理
Namesrv就是RMQ中的路由服务，可以类比ZK在Kafka中的作用，实现了路由管理、服务注册、服务发现等功能;不过Namesrv相比zookeeper来得要更轻便一点。
一、功能
Namesrv的功能大概可以总结为下面2点：
- 接收broker的请求注册broke路由信息（包括master和slave）：broker启动时向所有NameSrv注册，Producer在发送消息前先从Namesrv获取Broker服务器地址列表，通过负载算法选择对应服务器进行消息发送。Namesrv与每台broker保持长连接并进行间隔30s的心跳检测，broker宕机则从路由注册表中删除。
- 接收client的请求根据某个topic获取所有到broker的路由信息
二、启动流程
step1:解析配置文件，填充NameServerConfig,NettyServerConfig属性值
final NamesrvConfig namesrvConfig = new NamesrvConfig();
//这一行是在我本机运行源码demo所设置的RocketmqHome的值
namesrvConfig.setRocketmqHome("F:/J.Howie/rocketmq");
final NettyServerConfig nettyServerConfig = new NettyServerConfig();
nettyServerConfig.setListenPort(9876);
if (commandLine.hasOption('c')) {
String file = commandLine.getOptionValue('c');
if (file != null) {
InputStream in = new BufferedInputStream(new FileInputStream(file));
properties = new Properties();
properties.load(in);
MixAll.properties2Object(properties, namesrvConfig);
MixAll.properties2Object(properties, nettyServerConfig);
namesrvConfig.setConfigStorePath(file);
System.out.printf("load config properties file OK, %s%n", file);
in.close();
}
}
if (commandLine.hasOption('p')) {
InternalLogger console = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_CONSOLE_NAME);
MixAll.printObjectProperties(console, namesrvConfig);
MixAll.printObjectProperties(console, nettyServerConfig);
System.exit(0);
}
MixAll.properties2Object(ServerUtil.commandLine2Properties(commandLine), namesrvConfig);
先创建NameServerConfig(Namesrv业务参数)，NettyServerConfig(Namesrv网络参数)，然后再启动时把指定的配置文件或启动命令中的选项值填充到对象中。
- -c configFile 通过-c 命令指定配置文件的路劲
- 使用 “-- 属性名 属性值”,例如 --listenPort 9876
Step2：根据启动属性创建NamesrvController实例，并初始化该实例(step1中的2官网config都是NamesrvController的属性)，NamesrvController实例即Namesrv核心控制器
public boolean initialize() {
this.kvConfigManager.load();
//创建NettyServer网络处理对象
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);
this.remotingExecutor =
Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(), new ThreadFactoryImpl("RemotingExecutorThread_"));
this.registerProcessor();
//心跳1：每隔10s扫描一次Broker，移除处于不激活状态的broker
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.routeInfoManager.scanNotActiveBroker();
}
}, 5, 10, TimeUnit.SECONDS);
//心跳2：每隔10min打印一次kv配置
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.kvConfigManager.printAllPeriodically();
}
}, 1, 10, TimeUnit.MINUTES);
return true;
}
Step3:注册JVM钩子函数并启动服务器，以便监听broker、消息生产者的网络请求
//提示：代码中如果使用线程池，可以使用JVM钩子函数（shutdownHook）能够优雅的关闭线程池，及时释放资源
Runtime.getRuntime().addShutdownHook(new ShutdownHookThread(log, new Callable() {
@Override
public Void call() throws Exception {
controller.shutdown();
return null;
}
}));
controller.start();
三、RouteInfoManager
Namesrv用来存储路由的基础信息都放在RouteInfoManager类中,RouteInfoManager类也可以看做是Namesrv的资源类，很多操作都是对此类中的数据进行实时更改：
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
private final static long BROKER_CHANNEL_EXPIRED_TIME = 1000 * 60 * 2;
private final ReadWriteLock lock = new ReentrantReadWriteLock();
private final HashMap> topicQueueTable;
private final HashMap brokerAddrTable;
private final HashMap brokerLiveTable;
private final HashMap/* Filter Server */> filterServerTable;
- topicQueueTable：Topic消息队列路由信息，消息发送时根据路由表进行负载均衡
- brokerAddrTable：Broker基础信息，包含brokerName、所属集群名称、主备Broker地址
- clusterAddrTable：Broke集群信息，存储集群中所有Broker名称
- brokerLiveTable：Broker状态信息，NameServer每次收到心跳包时会替换该信息
- filterServerTable：Broker上的FilterServer列表，用于类模式消息过滤
QueueData、BrokerData、BrokerLiveInfo的类图：
运行时结构图：
TopicQueueTable、BrokerAddrTable运行时内存结构：
BrokerLiveTable、ClusterAddrTable运行时内存结构：
四、Namesrv与broker间的心跳
RMQ路由注册时通过Broker与Namesrv的心跳功能实现的，Broker启动10s后每间隔30s向集群中所有的Namesrv发送心跳包，Namesrv会根据收到的心跳包更新brokerLiveTable缓存中BrokerLiveInfo的lastUpdateTimestamp，然后Namesrv每隔10s扫描brokerLiveTable，如果连续120s没有收到心跳包，则踢除该broker并关闭socket连接。
4.1、Broker发送心跳包：
Broker端心跳包发送（BrokerController#start）
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
} catch (Throwable e) {
log.error("registerBrokerAll Exception", e);
}
}
}, 1000 * 10, Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS);
4.2、Namesrv处理心跳包
step1：添加写锁，防止并发修改路由表，然后判断该broker集群是否存在，不存在则创建并添加到broker集合中
this.lock.writeLock().lockInterruptibly();
Set brokerNames = this.clusterAddrTable.get(clusterName);
if (null == brokerNames) {
brokerNames = new HashSet();
this.clusterAddrTable.put(clusterName, brokerNames);
}
brokerNames.add(brokerName);
step2：维护BrokerData信息,先从brokerAdd人Table获取broker信息，不存在则新建BrokerData并放入到brokerAddrTable,registerFirst设置为true;如果存在，直接替换原先的，register设置为false，表示非第一次注册。
BrokerData brokerData = this.brokerAddrTable.get(brokerName);
if (null == brokerData) {
registerFirst = true;
brokerData = new BrokerData(clusterName, brokerName, new HashMap());
this.brokerAddrTable.put(brokerName, brokerData);
}
String oldAddr = brokerData.getBrokerAddrs().put(brokerId, brokerAddr);
registerFirst = registerFirst || (null == oldAddr);
step3：如果broker为master,并且brokerTopic配置信息发生变化或者是初次注册，则需要创建或更新topic路由元数据并填充topicQueueTable。
if (null != topicConfigWrapper
&& MixAll.MASTER_ID == brokerId) {
if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
|| registerFirst) {
ConcurrentMap tcTable =
topicConfigWrapper.getTopicConfigTable();
if (tcTable != null) {
for (Map.Entry entry : tcTable.entrySet()) {
this.createAndUpdateQueueData(brokerName, entry.getValue());
}
}
}
}
step4：更新BrokerLiveInfo,存活Broker信息表,BrokerLiveInfo是执行路由删除的重要依据。
BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
new BrokerLiveInfo(
System.currentTimeMillis(),
topicConfigWrapper.getDataVersion(),
channel,
haServerAddr));
if (null == prevBrokerLiveInfo) {
log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
}
step5：注册broker的过滤器Server地址列表,一个Broker上会关联多个FilterServer消息过滤服务器;如果此broker为从节点，则需要查找该broker的master节点信息，并更新对应的masterAddress属性。
if (filterServerList != null) {
if (filterServerList.isEmpty()) {
this.filterServerTable.remove(brokerAddr);
} else {
this.filterServerTable.put(brokerAddr, filterServerList);
}
}
if (MixAll.MASTER_ID != brokerId) {
String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
if (masterAddr != null) {
BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
if (brokerLiveInfo != null) {
result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
result.setMasterAddr(masterAddr);
}
}
}
五、其他功能
上面第四章介绍了broker和Namesrv的心跳机制，也可以看做是Namesrv的路由注册功能。下面介绍下Namesrv的其他功能：
5.1、路由删除
上文提到，Broker每隔30s向Namesrv发送一个心跳包,Namesrv会每隔10s扫描brokerLiveTable状态表,如果BrokerLive的lastUpdateTimestamp的时间戳距当前时间超过120s，则认为broker失效并移除关闭，同时更新其他状态信息。
路由删除会从topicQueueTable、brokerAddrTable、brokerLiveTable、filterServerTable中删除与该Broker相关的信息
public void onChannelDestroy(String remoteAddr, Channel channel) {
String brokerAddrFound = null;
if (brokerAddrFound != null && brokerAddrFound.length() > 0) {
try {
try {
//1、申请写锁，根据brokerAddress从brokerLiveTable、filterServerTable移除
this.lock.writeLock().lockInterruptibly();
this.brokerLiveTable.remove(brokerAddrFound);
this.filterServerTable.remove(brokerAddrFound);
//2、维护brokerAddrTable,遍历brokerAdd人Table，从其的BrokerData中的brokerAddres找到具体的Broker，从BrokerData中移除
String brokerNameFound = null;
boolean removeBrokerName = false;
Iterator> itBrokerAddrTable =
this.brokerAddrTable.entrySet().iterator();
while (itBrokerAddrTable.hasNext() && (null == brokerNameFound)) {
BrokerData brokerData = itBrokerAddrTable.next().getValue();
Iterator> it = brokerData.getBrokerAddrs().entrySet().iterator();
while (it.hasNext()) {
Entry entry = it.next();
Long brokerId = entry.getKey();
String brokerAddr = entry.getValue();
if (brokerAddr.equals(brokerAddrFound)) {
brokerNameFound = brokerData.getBrokerName();
it.remove();
log.info("remove brokerAddr[{}, {}] from brokerAddrTable, because channel destroyed",
brokerId, brokerAddr);
break;
}
}
if (brokerData.getBrokerAddrs().isEmpty()) {
removeBrokerName = true;
itBrokerAddrTable.remove();
log.info("remove brokerName[{}] from brokerAddrTable, because channel destroyed",
brokerData.getBrokerName());
}
}
//3、根据BrokerName,从ClusterAddrTable中找到Broker并从集群中移除，
//如果移除后，集群中不包含任何Broker,则将该集群从clusterAddrTable中移除
if (brokerNameFound != null && removeBrokerName) {
Iterator>> it = this.clusterAddrTable.entrySet().iterator();
while (it.hasNext()) {
Entry> entry = it.next();
String clusterName = entry.getKey();
Set brokerNames = entry.getValue();
boolean removed = brokerNames.remove(brokerNameFound);
if (removed) {
log.info("remove brokerName[{}], clusterName[{}] from clusterAddrTable, because channel destroyed",
brokerNameFound, clusterName);
if (brokerNames.isEmpty()) {
log.info("remove the clusterName[{}] from clusterAddrTable, because channel destroyed and no broker in this cluster",
clusterName);
it.remove();
}
break;
}
}
}
//4、根据brokerName，遍历所有主题的队列，
if (removeBrokerName) {
Iterator>> itTopicQueueTable =
this.topicQueueTable.entrySet().iterator();
while (itTopicQueueTable.hasNext()) {
Entry> entry = itTopicQueueTable.next();
String topic = entry.getKey();
List queueDataList = entry.getValue();
Iterator itQueueData = queueDataList.iterator();
while (itQueueData.hasNext()) {
QueueData queueData = itQueueData.next();
if (queueData.getBrokerName().equals(brokerNameFound)) {
itQueueData.remove();
log.info("remove topic[{} {}], from topicQueueTable, because channel destroyed",
topic, queueData);
}
}
if (queueDataList.isEmpty()) {
itTopicQueueTable.remove();
log.info("remove topic[{}] all queue, from topicQueueTable, because channel destroyed",
topic);
}
}
}
} finally {
//5、释放锁
this.lock.writeLock().unlock();
}
} catch (Exception e) {
log.error("onChannelDestroy Exception", e);
}
}
}
5.2、路由发现
RMQ路由发现是非实时的，当Topic发生变化后，Namesrv不知道推送给consumer，而是有consumer拉取最新的路由。
public RemotingCommand getRouteInfoByTopic(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
final RemotingCommand response = RemotingCommand.createResponseCommand(null);
final GetRouteInfoRequestHeader requestHeader =
(GetRouteInfoRequestHeader) request.decodeCommandCustomHeader(GetRouteInfoRequestHeader.class);
//1、调用RouteInfoManager的方法，填充路由表中的值
TopicRouteData topicRouteData = this.namesrvController.getRouteInfoManager().pickupTopicRouteData(requestHeader.getTopic());
//2、找到主题对应的路由信息并且该主题为顺序消息，则从NameServerKVconfig中获取关于顺序消息相关的配置填充路由信息；如果没找到路由信息则使用TOPIC_NOT_EXISTS
if (topicRouteData != null) {
if (this.namesrvController.getNamesrvConfig().isOrderMessageEnable()) {
String orderTopicConf =
this.namesrvController.getKvConfigManager().getKVConfig(NamesrvUtil.NAMESPACE_ORDER_TOPIC_CONFIG,
requestHeader.getTopic());
topicRouteData.setOrderTopicConf(orderTopicConf);
}
byte[] content = topicRouteData.encode();
response.setBody(content);
response.setCode(ResponseCode.SUCCESS);
response.setRemark(null);
return response;
}
response.setCode(ResponseCode.TOPIC_NOT_EXIST);
response.setRemark("No topic route info in name server for the topic: " + requestHeader.getTopic()
+ FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL));
return response;
}
参考资料：
RocketMQ核心技术
RocketMQ实战与原理解析
博客