路由元信息
之前我们分析了NameServer的源码和功能，提到了这次分析一下路由发现、注册、剔除功能的源码，我们直入正题，先看下路由的元信息，在RouteInfoManager类中保存：
private final HashMap> topicQueueTable;
private final HashMap brokerAddrTable;
private final HashMap brokerLiveTable;
private final HashMap/* Filter Server */> filterServerTable;
- topicQueueTable：topic的元信息，消息发送时会根据路由表进行负载均衡
- brokerAddrTable：broker的元信息，包含brokerName、所属集群名称、主备broker地址
- clusterAddrTable：集群的云信息
- brokerLiveTable：broker是否存活的状态信息
-
filterServerTable：filterServer列表，用于类模式消息过滤
元数据
路由注册
路由注册发起者肯定是broker端，在BrokerController中可以找到如下代码：
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.registerBrokerAll(true, false, BrokerController.this.brokerConfig.isForceRegister());
} catch (final Throwable e) {
log.error("registerBrokerAll Exception", e);
}
}
}, 1000 * 10, Math.max(10000, Math.min(this.brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS);
定时任务会每隔30秒向NameServer注册broker信息，该注册方法最终会调用到BrokerOuterAPI的registerBrokerAll()方法
public List registerBrokerAll(
final String clusterName,
final String brokerAddr,
final String brokerName,
final long brokerId,
final String haServerAddr,
final TopicConfigSerializeWrapper topicConfigWrapper,
final List filterServerList,
final boolean oneway,
final int timeoutMills,
final boolean compressed) {
final List registerBrokerResultList = Lists.newArrayList();
List nameServerAddressList = this.remotingClient.getNameServerAddressList();
if (nameServerAddressList != null && nameServerAddressList.size() > 0) {
final RegisterBrokerRequestHeader requestHeader = new RegisterBrokerRequestHeader();
requestHeader.setBrokerAddr(brokerAddr);
requestHeader.setBrokerId(brokerId);
requestHeader.setBrokerName(brokerName);
requestHeader.setClusterName(clusterName);
requestHeader.setHaServerAddr(haServerAddr);
requestHeader.setCompressed(compressed);
RegisterBrokerBody requestBody = new RegisterBrokerBody();
requestBody.setTopicConfigSerializeWrapper(topicConfigWrapper);
requestBody.setFilterServerList(filterServerList);
final byte[] body = requestBody.encode(compressed);
final int bodyCrc32 = UtilAll.crc32(body);
requestHeader.setBodyCrc32(bodyCrc32);
final CountDownLatch countDownLatch = new CountDownLatch(nameServerAddressList.size());
for (final String namesrvAddr : nameServerAddressList) {
brokerOuterExecutor.execute(new Runnable() {
@Override
public void run() {
try {
RegisterBrokerResult result = registerBroker(namesrvAddr,oneway, timeoutMills,requestHeader,body);
if (result != null) {
registerBrokerResultList.add(result);
}
log.info("register broker to name server {} OK", namesrvAddr);
} catch (Exception e) {
log.warn("registerBroker Exception, {}", namesrvAddr, e);
} finally {
countDownLatch.countDown();
}
}
});
}
try {
countDownLatch.await(timeoutMills, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
}
}
return registerBrokerResultList;
}
该方法会遍历所有NameServer注册，然后会由RouteInfoManager的registerBroker()方法完成注册。
public RegisterBrokerResult registerBroker(
final String clusterName,
final String brokerAddr,
final String brokerName,
final long brokerId,
final String haServerAddr,
final TopicConfigSerializeWrapper topicConfigWrapper,
final List filterServerList,
final Channel channel) {
RegisterBrokerResult result = new RegisterBrokerResult();
try {
try {
// 写锁
this.lock.writeLock().lockInterruptibly();
Set brokerNames = this.clusterAddrTable.get(clusterName);
if (null == brokerNames) {
brokerNames = new HashSet();
this.clusterAddrTable.put(clusterName, brokerNames);
}
brokerNames.add(brokerName);
// 是否第一次注册
boolean registerFirst = false;
// 获取并维护broker信息，如果首次注册，就新增broker信息
BrokerData brokerData = this.brokerAddrTable.get(brokerName);
if (null == brokerData) {
registerFirst = true;
brokerData = new BrokerData(clusterName, brokerName, new HashMap());
this.brokerAddrTable.put(brokerName, brokerData);
}
String oldAddr = brokerData.getBrokerAddrs().put(brokerId, brokerAddr);
registerFirst = registerFirst || (null == oldAddr);
// 如果是master，并且topic信息发生变化或者是初次注册，则需要创建topic或者修改topic信息
if (null != topicConfigWrapper
&& MixAll.MASTER_ID == brokerId) {
if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
|| registerFirst) {
ConcurrentMap tcTable =
topicConfigWrapper.getTopicConfigTable();
if (tcTable != null) {
for (Map.Entry entry : tcTable.entrySet()) {
this.createAndUpdateQueueData(brokerName, entry.getValue());
}
}
}
}
// 更新BrokerLiveInfo信息，broker存活信息表，路由剔除会根据它进行剔除
BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
new BrokerLiveInfo(
System.currentTimeMillis(),
topicConfigWrapper.getDataVersion(),
channel,
haServerAddr));
if (null == prevBrokerLiveInfo) {
log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
}
// broker的filterServer列表
if (filterServerList != null) {
if (filterServerList.isEmpty()) {
this.filterServerTable.remove(brokerAddr);
} else {
this.filterServerTable.put(brokerAddr, filterServerList);
}
}
// 如果是从节点，则需要根据主节点的信息做一些ha设置
if (MixAll.MASTER_ID != brokerId) {
String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
if (masterAddr != null) {
BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
if (brokerLiveInfo != null) {
result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
result.setMasterAddr(masterAddr);
}
}
}
} finally {
this.lock.writeLock().unlock();
}
} catch (Exception e) {
log.error("registerBroker Exception", e);
}
return result;
}
路由剔除
RocketMQ有两种方式触发路由剔除
- NameServer定时扫描brokerLiveTable检测上次心跳与当前时间的时间差，如果大于两分钟，则需要剔除该Broker信息
- Broker在正常关闭的情况下，会执行unregisterBroker指令
每隔10s，会执行RouteInfoManager的scanNotActiveBroker()方法，该方法会扫描broker，移除不处于激活状态的broker。
public void scanNotActiveBroker() {
final Iterator> it = this.brokerLiveTable.entrySet().iterator();
while (it.hasNext()) {
final Entry next = it.next();
final long last = next.getValue().getLastUpdateTimestamp();
if ((last + BROKER_CHANNEL_EXPIRED_TIME) < System.currentTimeMillis()) {
RemotingUtil.closeChannel(next.getValue().getChannel());
it.remove();
log.warn("The broker channel expired, {} {}ms", next.getKey(), BROKER_CHANNEL_EXPIRED_TIME);
this.onChannelDestroy(next.getKey(), next.getValue().getChannel());
}
}
}
扫描到上一次心跳时间和当前时间的时间差超过了两分钟，就移除broker，关闭channel，删除逻辑主要在onChannelDestroy()方法中
public void onChannelDestroy(final String remoteAddr, final Channel channel) {
String brokerAddrFound = null;
if (channel != null) {
try {
try {
this.lock.readLock().lockInterruptibly();
final Iterator> itBrokerLiveTable =
this.brokerLiveTable.entrySet().iterator();
while (itBrokerLiveTable.hasNext()) {
final Entry entry = itBrokerLiveTable.next();
if (entry.getValue().getChannel() == channel) {
brokerAddrFound = entry.getKey();
break;
}
}
} finally {
this.lock.readLock().unlock();
}
} catch (final Exception e) {
log.error("onChannelDestroy Exception", e);
}
}
if (null == brokerAddrFound) {
brokerAddrFound = remoteAddr;
} else {
log.info("the broker's channel destroyed, {}, clean it's data structure at once", brokerAddrFound);
}
if (brokerAddrFound != null && brokerAddrFound.length() > 0) {
try {
try {
this.lock.writeLock().lockInterruptibly();
this.brokerLiveTable.remove(brokerAddrFound);
this.filterServerTable.remove(brokerAddrFound);
String brokerNameFound = null;
boolean removeBrokerName = false;
// 从brokerAddrTable移除该broker
final Iterator> itBrokerAddrTable =
this.brokerAddrTable.entrySet().iterator();
while (itBrokerAddrTable.hasNext() && (null == brokerNameFound)) {
final BrokerData brokerData = itBrokerAddrTable.next().getValue();
final Iterator> it = brokerData.getBrokerAddrs().entrySet().iterator();
while (it.hasNext()) {
final Entry entry = it.next();
final Long brokerId = entry.getKey();
final String brokerAddr = entry.getValue();
if (brokerAddr.equals(brokerAddrFound)) {
brokerNameFound = brokerData.getBrokerName();
it.remove();
log.info("remove brokerAddr[{}, {}] from brokerAddrTable, because channel destroyed",
brokerId, brokerAddr);
break;
}
}
if (brokerData.getBrokerAddrs().isEmpty()) {
removeBrokerName = true;
itBrokerAddrTable.remove();
log.info("remove brokerName[{}] from brokerAddrTable, because channel destroyed",
brokerData.getBrokerName());
}
}
// 从clusterAddrTable移除该broker
if (brokerNameFound != null && removeBrokerName) {
final Iterator>> it = this.clusterAddrTable.entrySet().iterator();
while (it.hasNext()) {
final Entry> entry = it.next();
final String clusterName = entry.getKey();
final Set brokerNames = entry.getValue();
final boolean removed = brokerNames.remove(brokerNameFound);
if (removed) {
log.info("remove brokerName[{}], clusterName[{}] from clusterAddrTable, because channel destroyed",
brokerNameFound, clusterName);
if (brokerNames.isEmpty()) {
log.info("remove the clusterName[{}] from clusterAddrTable, because channel destroyed and no broker in this cluster",
clusterName);
it.remove();
}
break;
}
}
}
// 从topicQueueTable移除包含该broker的topic 信息
if (removeBrokerName) {
final Iterator>> itTopicQueueTable =
this.topicQueueTable.entrySet().iterator();
while (itTopicQueueTable.hasNext()) {
final Entry> entry = itTopicQueueTable.next();
final String topic = entry.getKey();
final List queueDataList = entry.getValue();
final Iterator itQueueData = queueDataList.iterator();
while (itQueueData.hasNext()) {
final QueueData queueData = itQueueData.next();
if (queueData.getBrokerName().equals(brokerNameFound)) {
itQueueData.remove();
log.info("remove topic[{} {}], from topicQueueTable, because channel destroyed",
topic, queueData);
}
}
if (queueDataList.isEmpty()) {
itTopicQueueTable.remove();
log.info("remove topic[{}] all queue, from topicQueueTable, because channel destroyed",
topic);
}
}
}
} finally {
this.lock.writeLock().unlock();
}
} catch (final Exception e) {
log.error("onChannelDestroy Exception", e);
}
}
}
这个方法看上去很长，主要就是把元数据信息里面关于这个broker信息删除掉
路由发现
DefaultRequestProcessor类中getRouteInfoByTopic()方法会调用RouteInfoManager的pickupTopicRouteData()处理请求，从路由表topicQueueTable、brokerAddrTable、filterServerTable分别填充到TopicTouteData，TopicTouteData的信息如下：
private String orderTopicConf;
// 队列元数据
private List queueDatas;
// topic 分布的broker元数据
private List brokerDatas;
// broker 上过滤服务器地址列表
private HashMap/* Filter Server */> filterServerTable;
接下来我们看下pickupTopicRouteData()方法
public TopicRouteData pickupTopicRouteData(final String topic) {
final TopicRouteData topicRouteData = new TopicRouteData();
boolean foundQueueData = false;
boolean foundBrokerData = false;
final Set brokerNameSet = new HashSet<>();
final List brokerDataList = new LinkedList<>();
topicRouteData.setBrokerDatas(brokerDataList);
final HashMap> filterServerMap = new HashMap<>();
topicRouteData.setFilterServerTable(filterServerMap);
try {
try {
this.lock.readLock().lockInterruptibly();
final List queueDataList = this.topicQueueTable.get(topic);
if (queueDataList != null) {
topicRouteData.setQueueDatas(queueDataList);
foundQueueData = true;
final Iterator it = queueDataList.iterator();
while (it.hasNext()) {
final QueueData qd = it.next();
brokerNameSet.add(qd.getBrokerName());
}
for (final String brokerName : brokerNameSet) {
final BrokerData brokerData = this.brokerAddrTable.get(brokerName);
if (null != brokerData) {
final BrokerData brokerDataClone = new BrokerData(brokerData.getCluster(), brokerData.getBrokerName(), (HashMap) brokerData
.getBrokerAddrs().clone());
brokerDataList.add(brokerDataClone);
foundBrokerData = true;
for (final String brokerAddr : brokerDataClone.getBrokerAddrs().values()) {
final List filterServerList = this.filterServerTable.get(brokerAddr);
filterServerMap.put(brokerAddr, filterServerList);
}
}
}
}
} finally {
this.lock.readLock().unlock();
}
} catch (final Exception e) {
log.error("pickupTopicRouteData Exception", e);
}
log.debug("pickupTopicRouteData {} {}", topic, topicRouteData);
if (foundBrokerData && foundQueueData) {
return topicRouteData;
}
return null;
}
该方法从路由表topicQueueTable、brokerAddrTable、filterServerTable分别填充到TopicTouteData并返回
小结
- DefaultRequestProcessor类会处理各种有关路由信息的请求，路由的元信息保存在RouteInfoManager类中。
- Broker每30s会向NameServer注册自己。
- NameServer每隔10s 会扫描并移除不可用的Broker。