RocketMQ源码系列(二)RocketMQ路由中心,NameServer源码篇,路由注册,路由发现,路由删除

文章目录

  • 1、RocketMQ设计理念
  • 2、RocketMQ解决的问题
  • 3、NameServer所承担的作用
  • 4、NameServer启动流程
    • 入口main函数
    • 创建Namesrv核心控制器
    • start(controller)方法
    • controller.initialize()初始化方法
    • NamesrvConfig属性
    • NettyServerConfig属性
  • 5、NameServer路由元信息
    • NameServer存储的信息
  • 6、NameServer路由注册
    • broker发送心跳包
    • NameServer处理心跳包
      • 维护clusterAddrTable
      • 维护brokerAddrTable
      • 维护topicQueueTable
      • 维护brokerLiveTable
      • 维护filterServerTable
    • 路由注册总结
  • 7、路由删除
    • 定时删除路由信息
    • scanNotActiveBroker方法
    • onChannelDestroy方法逻辑如下
  • 8、路由发现

1、RocketMQ设计理念

RocketMQ基于主题的发布与订阅模式,核心包括消息发送、消息存储、消息消费。自研NameServer实现元数据的管理,设计简单,并没有采用Zookeeper作为注册中心。集群之间不保持强一致,追求最终一致性,能容忍分钟级的不一致,所以集群之间互不通信,降低了NameServer的复杂性,网络要求也降低,性能比Zookeeper提高不少。高效的IO存储机制,以及存储分组,组内单个文件大小固定,引入内存映射机制,所有消息基于顺序写,极大地提升写性能。同时引入消息消费队列文件和索引文件,提升了消息消费和查找的效率。

2、RocketMQ解决的问题

顺序消息、消息存储、消息高可用、消息消费低延迟、确保至少被消费一次、回溯消息、消息堆积、定时消息、消息重试等。

3、NameServer所承担的作用

消息中间件一般的设计思路是基于主题的发布订阅机制,消息生产者发送某一主题消息到消息服务器,消息服务器负责消息的持久化存储,消费者订阅感兴趣的主题,消息服务器根据订阅信息将消息推送给消费者,或者消费者主动向消息服务器拉取消息,从而实现生产者与消费者的解耦。那么消息生产者如何知道消息要发往那台消息服务器呢?以及如果有一台消息服务器宕机了,生产者怎么在不重启服务的情况下感知到呢?NameServer就是为了解决这些问题而生。

4、NameServer启动流程

首先打开NameServer启动类NamesrvStartup
RocketMQ源码系列(二)RocketMQ路由中心,NameServer源码篇,路由注册,路由发现,路由删除_第1张图片

入口main函数

启动方法我们可以看到入口函数内容封装成了main0(args)方法,根据参数组装得到NamesrvController对象,然后调用启动方法。
启动方法执行完成后打印日志输出启动成功的信息。

    public static void main(String[] args) {
     
        main0(args);
    }

    public static NamesrvController main0(String[] args) {
     

        try {
     
            //根据参数组装得到NamesrvController对象
            NamesrvController controller = createNamesrvController(args);
            start(controller);//调用启动方法
            String tip = "The Name Server boot success. serializeType=" + RemotingCommand.getSerializeTypeConfigInThisServer();
            log.info(tip);
            System.out.printf("%s%n", tip);
            return controller;
        } catch (Throwable e) {
     
            e.printStackTrace();
            System.exit(-1);
        }

        return null;
    }

创建Namesrv核心控制器

通过createNamesrvController(args)方法,创建Namesrv核心控制器

    public static NamesrvController createNamesrvController(String[] args) throws IOException, JoranException {
     
        //设置key、value键值对的系统属性,添加rocketmq当前版本
        System.setProperty(RemotingCommand.REMOTING_VERSION_KEY, Integer.toString(MQVersion.CURRENT_VERSION));
        //PackageConflictDetect.detectFastjson();

        //构建命令行操作的指令
        Options options = ServerUtil.buildCommandlineOptions(new Options());
        //得到启动namesrv的命令
        commandLine = ServerUtil.parseCmdLine("mqnamesrv", args, buildCommandlineOptions(options), new PosixParser());
        if (null == commandLine) {
     
            System.exit(-1);
            return null;
        }

        //创建namesrv业务参数
        final NamesrvConfig namesrvConfig = new NamesrvConfig();
        //创建namesrv网格参数
        final NettyServerConfig nettyServerConfig = new NettyServerConfig();
        nettyServerConfig.setListenPort(9876);//设置监听端口
        /*
         在解析启动时把指定的配置文件或启动命令中的选项值,
         填充到namesrvConfig,nettyServerConfig对象中,
         参数的来源主要两种方式:
         1)-c configFile 通过-c命令指定配置文件的路径
         2)使用“--属性名 属性值”的方式入的参,例如--listenPort 9876
         */
        if (commandLine.hasOption('c')) {
     
            //如果启动命令指定了配置文件,获取配置文件路径,读取文件内容
            String file = commandLine.getOptionValue('c');
            if (file != null) {
     
                InputStream in = new BufferedInputStream(new FileInputStream(file));
                properties = new Properties();
                properties.load(in);
                MixAll.properties2Object(properties, namesrvConfig);
                MixAll.properties2Object(properties, nettyServerConfig);

                //设置命令行启动namesrv指定的配置文件路径
                namesrvConfig.setConfigStorePath(file);

                System.out.printf("load config properties file OK, %s%n", file);
                in.close();
            }
        }

        //控制台打印namesrv的配置信息,命令行上面加p
        if (commandLine.hasOption('p')) {
     
            InternalLogger console = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_CONSOLE_NAME);
            MixAll.printObjectProperties(console, namesrvConfig);
            MixAll.printObjectProperties(console, nettyServerConfig);
            System.exit(0);
        }

        //把命令行属性解析成properties
        MixAll.properties2Object(ServerUtil.commandLine2Properties(commandLine), namesrvConfig);

        //判断是否设置rocketmqHome。没有设置则报错退出
        if (null == namesrvConfig.getRocketmqHome()) {
     
            System.out.printf("Please set the %s variable in your environment to match the location of the RocketMQ installation%n", MixAll.ROCKETMQ_HOME_ENV);
            System.exit(-2);
        }

        //加载rocketmq日志对象
        LoggerContext lc = (LoggerContext) LoggerFactory.getILoggerFactory();
        JoranConfigurator configurator = new JoranConfigurator();
        configurator.setContext(lc);
        lc.reset();
        /*
          读取日志配置文件,此处说明我们的日志配置文件必须是
          相对于指定的rocketmqHome下的conf/logback_namesrv.xml
          名字固定
        */
        configurator.doConfigure(namesrvConfig.getRocketmqHome() + "/conf/logback_namesrv.xml");

        log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);

        MixAll.printObjectProperties(log, namesrvConfig);
        MixAll.printObjectProperties(log, nettyServerConfig);
        //根据namesrvConfig和nettyServerConfig初始化controller,final不可变更
        final NamesrvController controller = new NamesrvController(namesrvConfig, nettyServerConfig);

        // remember all configs to prevent discard
        //注册配置文件
        controller.getConfiguration().registerConfig(properties);

        return controller;
    }

start(controller)方法

如下,首先对参数进行初始化,然后在jvm中增加一个关闭的钩子,当jvm关闭的时候,会执行系统中已经设置的所有通过方法addShutdownHook添加的钩子,此处当系统执行完controller.shutdown()后,jvm才会关闭。然后调用启动方法。

    public static NamesrvController start(final NamesrvController controller) throws Exception {
     

        if (null == controller) {
     
            throw new IllegalArgumentException("NamesrvController is null");
        }

        boolean initResult = controller.initialize();//对参数进行初始化
        if (!initResult) {
     //初始化失败则退出
            controller.shutdown();
            System.exit(-3);
        }

        /*在jvm中增加一个关闭的钩子,当jvm关闭的时候,
          会执行系统中已经设置的所有通过方法addShutdownHook添加的钩子,
          当系统执行完controller.shutdown()后,jvm才会关闭。*/
        Runtime.getRuntime().addShutdownHook(new ShutdownHookThread(log, new Callable<Void>() {
     
            @Override
            public Void call() throws Exception {
     
                controller.shutdown();
                return null;
            }
        }));

        controller.start();//调用启动方法

        return controller;
    }

controller.initialize()初始化方法

    public boolean initialize() {
     
        //加载KV配置
        this.kvConfigManager.load();
        //创建NettyServer网络处理对象
        this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);
        //创建处理网络连接的线程池
        this.remotingExecutor =
            Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(), new ThreadFactoryImpl("RemotingExecutorThread_"));
        //注册业务处理器
        this.registerProcessor();
        //开启删除不活跃broker的定时任务,间隔10s扫描一次,移除不活跃broker
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
     

            @Override
            public void run() {
     
                NamesrvController.this.routeInfoManager.scanNotActiveBroker();
            }
        }, 5, 10, TimeUnit.SECONDS);
        //启动一个定时任务,每十分钟打印一次KV配置
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
     

            @Override
            public void run() {
     
                NamesrvController.this.kvConfigManager.printAllPeriodically();
            }
        }, 1, 10, TimeUnit.MINUTES);

        if (TlsSystemConfig.tlsMode != TlsMode.DISABLED) {
     
            // Register a listener to reload SslContext
            // 注册一个监听器去重新加载SslContext
            try {
     
                fileWatchService = new FileWatchService(
                    new String[] {
     
                        TlsSystemConfig.tlsServerCertPath,
                        TlsSystemConfig.tlsServerKeyPath,
                        TlsSystemConfig.tlsServerTrustCertPath
                    },
                    new FileWatchService.Listener() {
     
                        boolean certChanged, keyChanged = false;
                        @Override
                        public void onChanged(String path) {
     
                            if (path.equals(TlsSystemConfig.tlsServerTrustCertPath)) {
     
                                log.info("The trust certificate changed, reload the ssl context");
                                reloadServerSslContext();
                            }
                            if (path.equals(TlsSystemConfig.tlsServerCertPath)) {
     
                                certChanged = true;
                            }
                            if (path.equals(TlsSystemConfig.tlsServerKeyPath)) {
     
                                keyChanged = true;
                            }
                            if (certChanged && keyChanged) {
     
                                log.info("The certificate and private key changed, reload the ssl context");
                                certChanged = keyChanged = false;
                                reloadServerSslContext();
                            }
                        }
                        private void reloadServerSslContext() {
     
                            ((NettyRemotingServer) remotingServer).loadSslContext();
                        }
                    });
            } catch (Exception e) {
     
                log.warn("FileWatchService created error, can't load the certificate dynamically");
            }
        }

        return true;
    }

NamesrvConfig属性

public class NamesrvConfig {
     
    private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
    /*
     * rocketmqHome表示主目录,可以通过如下两种方式配置RocketMQ的主目录
     * 1)-Drocketmq.home.dir=path
     * 2)通过设置环境变量ROCKETMQ_HOME
     */
    private String rocketmqHome = System.getProperty(MixAll.ROCKETMQ_HOME_PROPERTY, System.getenv(MixAll.ROCKETMQ_HOME_ENV));
    //NameServer存储KV配置属性的持久化路径。
    private String kvConfigPath = System.getProperty("user.home") + File.separator + "namesrv" + File.separator + "kvConfig.json";
    /*
     * 默认配置文件路径,但是不生效。nameServer启动时,
     * 如果要通过配置文件配置NameServer启动属性的话,
     * 要使用-c选项指定配置文件。
     */
    private String configStorePath = System.getProperty("user.home") + File.separator + "namesrv" + File.separator + "namesrv.properties";
    //集群测试使用的参数
    private String productEnvName = "center";
    //是否是集群测试,默认为false
    private boolean clusterTest = false;
    //是否支持顺序消息,默认是不支持。
    private boolean orderMessageEnable = false;

NettyServerConfig属性

public class NettyServerConfig implements Cloneable {
     
    //nameserver监听端口,该值会默认被初始化为9876
    private int listenPort = 8888;
    //Netty业务线程池线程个数
    private int serverWorkerThreads = 8;
    /*
     Netty public任务线程池线程个数,Netty网络设计会根据业务类型创建不同的线程池
     如消息发送、消息消费、心跳检测等业务场景,如果某个业务类型未注册到线程池,
     则由public线程池执行。
     */
    private int serverCallbackExecutorThreads = 0;
    /*
     IO线程池个数,主要是NameServer、Broker端解析请求、返回相应的线程个数,这类线程
     主要是处理网络请求的,解析请求包,然后转发到各个业务线程池完成具体业务操作,
     再将结果返回给调用方。
     */
    private int serverSelectorThreads = 3;
    //send oneway类型的消息请求并发度(Broker端参数)
    private int serverOnewaySemaphoreValue = 256;
    //异步消息请求最大并发度(Broker端参数)
    private int serverAsyncSemaphoreValue = 64;
    /*
     网络连接最大空闲时间,默认120s,如果连接空闲时间超过该参数设置的值,连接将被关闭。
     */
    private int serverChannelMaxIdleTimeSeconds = 120;
    //网络socket发送缓存区大小,默认是64k
    private int serverSocketSndBufSize = NettySystemConfig.socketSndbufSize;
    //网络socket接收缓存区大小,默认是64k
    private int serverSocketRcvBufSize = NettySystemConfig.socketRcvbufSize;
    //ByteBuffer是否开启缓存,建议是开启
    private boolean serverPooledByteBufAllocatorEnable = true;

    /**
     * make make install
     * 是否启用Epll IO模型,Linux环境建议开启
     *
     * ../glibc-2.10.1/configure \ --prefix=/usr \ --with-headers=/usr/include \
     * --host=x86_64-linux-gnu \ --build=x86_64-pc-linux-gnu \ --without-gd
     */
    private boolean useEpollNativeSelector = false;

5、NameServer路由元信息

NameServer主要是为了给生产者和消费者提供topic的路由信息,那么NameServer需要存储那些信息呢?并且还要管理Broker节点,包括路由注册,路由删除等。

NameServer存储的信息

首先找到RouteInfoManager
RocketMQ源码系列(二)RocketMQ路由中心,NameServer源码篇,路由注册,路由发现,路由删除_第2张图片
代码解释如下

public class RouteInfoManager {
     
    private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
    private final static long BROKER_CHANNEL_EXPIRED_TIME = 1000 * 60 * 2;
    private final ReadWriteLock lock = new ReentrantReadWriteLock();
    //Topic消息队列路由信息,消息发送时根据路由表进行负载均衡
    private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
    //存储broker的基本信息,比如名称、所属集群、主备地址等
    private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
    //存储broker集群中所有broker名称,
    private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
    //broker状态表,每次nameserver收到心跳包时更新此表信息
    private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
    //broker上的filterServer列表,用于类模式消息过滤
    private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;

RocketMQ中一个Topic拥有多个消息队列,默认情况下一个broker为每一个topic创建4个读队列4个写队列。多个broker组成一个集群,brokerName由相同的多台broker组成Master-Slave架构,brokerId为0代表master,大于0表示Slave。BrokerLiveInfo中的lastUpdateTimestamp属性存储上次收到心跳包的时间。

6、NameServer路由注册

RocketMQ路由注册时通过broker与nameserver的心跳机制实现的。broker启动时向所有集群中的nameserver发送心跳请求,每隔30s向所有集群中的nameserver发送心跳包,nameserver收到broker心跳包时会更新brokerLiveTable缓存中brokerLiveInfo的lastUpdateTimestamp字段,然后nameserver每隔10s扫描brokerLiveTable,如果连续120s没有收到心跳包,nameserver将移除该broker的路由信息同时关闭socket连接。

broker发送心跳包

BrokerController的start方法中添加一个定时任务,执行BrokerController的registerBrokerAll方法,向所有的nameserver注册路由信息。

        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
     

            @Override
            public void run() {
     
                try {
     
                    BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
                } catch (Throwable e) {
     
                    log.error("registerBrokerAll Exception", e);
                }
            }
        }, 1000 * 10, Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS);

BrokerController的registerBrokerAll方法调用doRegisterBrokerAll方法,doRegisterBrokerAll调用BrokerOuterAPI的registerBrokerAll方法进行注册

start
——>registerBrokerAll(BrokerController)
——>doRegisterBrokerAll(BrokerController)
——>registerBrokerAll(BrokerOuterAPI)

registerBrokerAll(BrokerOuterAPI)方法遍历nameserver列表,依次向nameserver发送心跳包。

 			final CountDownLatch countDownLatch = new CountDownLatch(nameServerAddressList.size());
            //遍历nameserver列表,依次向nameserver发送心跳包。
            for (final String namesrvAddr : nameServerAddressList) {
     
                brokerOuterExecutor.execute(new Runnable() {
     
                    @Override
                    public void run() {
     
                        try {
     
                            RegisterBrokerResult result = registerBroker(namesrvAddr,oneway, timeoutMills,requestHeader,body);
                            if (result != null) {
     
                                registerBrokerResultList.add(result);
                            }

                            log.info("register broker[{}]to name server {} OK", brokerId, namesrvAddr);
                        } catch (Exception e) {
     
                            log.warn("registerBroker Exception, {}", namesrvAddr, e);
                        } finally {
     
                            countDownLatch.countDown();
                        }
                    }
                });
            }

            try {
     
                countDownLatch.await(timeoutMills, TimeUnit.MILLISECONDS);
            } catch (InterruptedException e) {
     
            }

最终调用registerBroker(BrokerOuterAPI)发送心跳包,底层传输基于Netty。

NameServer处理心跳包

RouteInfoManager的registerBroker处理心跳包

维护clusterAddrTable

首先加锁写防止并发修改路由表,判断该集群是否存在,获取该集群对应的broker,如果不存在则创建集群,将broker加入集群中

                //加锁写防止并发修改路由表。
                this.lock.writeLock().lockInterruptibly();

                //判断该集群是否存在,获取该集群对应的broker
                Set<String> brokerNames = this.clusterAddrTable.get(clusterName);
                if (null == brokerNames) {
     
                    brokerNames = new HashSet<String>();
                    //如果不存在则创建集群
                    this.clusterAddrTable.put(clusterName, brokerNames);
                }
                //将broker加入集群中
                brokerNames.add(brokerName);

维护brokerAddrTable

首先从brokerAddrTable根据BrokerName尝试获取Broker信息,如果不存在,则新建BrokerData对象并放入到brokerAddrTable中,把registerFirst设置为true;如果存在,直接替换原先的,registerFirst为false。

				//是否是第一次注册,false表示不是首次注册
                boolean registerFirst = false;

                //尝试从brokerAddrTable中获取BrokerData信息
                BrokerData brokerData = this.brokerAddrTable.get(brokerName);
                if (null == brokerData) {
     
                    //如果为空则表示首次注册registerFirst置为true
                    registerFirst = true;
                    //新建brokerdata对象
                    brokerData = new BrokerData(clusterName, brokerName, new HashMap<Long, String>());
                    //添加到brokerAddrTable中
                    this.brokerAddrTable.put(brokerName, brokerData);
                }

维护topicQueueTable

如果broker为master,并且topic信息发生变化或者是首次注册,则需要更新或者创建topic路由元数据,填入topicQueueTable,实际上就是为默认主题自动注册路由信息,其中包含MixAll.DEFAULT_TOPIC的路由信息。当生产者发送topic时,如果该topic未创建并且BrokerConfig的autoCreateTopicEnable为true时,将返回MixAll.DEFAULT_TOPIC的路由信息。

                if (null != topicConfigWrapper
                    && MixAll.MASTER_ID == brokerId) {
     
                    if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
                        || registerFirst) {
     
                        ConcurrentMap<String, TopicConfig> tcTable =
                            topicConfigWrapper.getTopicConfigTable();
                        if (tcTable != null) {
     
                            for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
     
                                //维护topicQueueTable
                                this.createAndUpdateQueueData(brokerName, entry.getValue());
                            }
                        }
                    }
                }
    private void createAndUpdateQueueData(final String brokerName, final TopicConfig topicConfig) {
     
        QueueData queueData = new QueueData();
        queueData.setBrokerName(brokerName);
        queueData.setWriteQueueNums(topicConfig.getWriteQueueNums());
        queueData.setReadQueueNums(topicConfig.getReadQueueNums());
        queueData.setPerm(topicConfig.getPerm());//设置队列读写权限
        queueData.setTopicSynFlag(topicConfig.getTopicSysFlag());//topic同步标记

        //判断topicQueueTable中是否存在topic对应的队列信息
        List<QueueData> queueDataList = this.topicQueueTable.get(topicConfig.getTopicName());
        if (null == queueDataList) {
     
            queueDataList = new LinkedList<QueueData>();
            //不存在则创建topic路由信息
            queueDataList.add(queueData);
            //并添加到topicQueueTable
            this.topicQueueTable.put(topicConfig.getTopicName(), queueDataList);
            log.info("new topic registered, {} {}", topicConfig.getTopicName(), queueData);
        } else {
     
            //如果存在则和最新的路由信息进行比对
            boolean addNewOne = true;
            //遍历topicQueueTable中的路由信息
            Iterator<QueueData> it = queueDataList.iterator();
            while (it.hasNext()) {
     
                QueueData qd = it.next();
                if (qd.getBrokerName().equals(brokerName)) {
     
                    //当brokerName相同,对应的queueData也相同时,则说明没有新加broker信息
                    if (qd.equals(queueData)) {
     
                        addNewOne = false;
                    } else {
     
                        //当brokerName相同,对应的queueData不同时,
                        // 说明broker信息发生变化,移除旧的信息
                        log.info("topic changed, {} OLD: {} NEW: {}", topicConfig.getTopicName(), qd,
                                queueData);
                        it.remove();
                    }
                }
            }
            //broker信息发生变化时,移除旧的信息后,添加新的信息
            if (addNewOne) {
     
                queueDataList.add(queueData);
            }
        }
    }

维护brokerLiveTable

更新brokerLiveTable存活信息表,brokerLiveTable是执行路由删除的重要依据。

				//添加新的信息返回旧的信息
                BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
                    new BrokerLiveInfo(
                        System.currentTimeMillis(),
                        topicConfigWrapper.getDataVersion(),
                        channel,
                        haServerAddr));
                if (null == prevBrokerLiveInfo) {
     
                    log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
                }

维护filterServerTable

一个broker会关联多个filterServer消息过滤器。

				if (filterServerList != null) {
     
                    if (filterServerList.isEmpty()) {
     
                        this.filterServerTable.remove(brokerAddr);
                    } else {
     
                        this.filterServerTable.put(brokerAddr, filterServerList);
                    }
                }

                //如果此broker不是master,则需要查找该broker对应的master,
                // 并更新对应的masterAddr
                if (MixAll.MASTER_ID != brokerId) {
     
                    String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
                    if (masterAddr != null) {
     
                        BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
                        if (brokerLiveInfo != null) {
     
                            result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
                            result.setMasterAddr(masterAddr);
                        }
                    }
                }

路由注册总结

broker与nameserver进行长连接,broker状态存储在brokerLiveTable中,nameserver每收到一个心跳包,将更新brokerLiveTable中关于broker的状态信息以及路由表(上面更新的几个table),更新时使用了锁粒度较少的读写锁,允许多个生产者并发读取,保证消息发送时的高并发。但同一时刻nameserver只处理一个broker心跳包,多个心跳包处理串行执行。这也是读写锁经典用法。

7、路由删除

当broker宕机,nameserver无法收到心跳包,此时nameserver如何来剔除这些失效的broker呢?nameserver会每隔10s扫描brokerLiveTable状态表,如果BrokerLive的lastUpdateTimestamp的时间戳距离当前时间超过120s,则认为broker失效,移除该broker,关闭与broker连接,并同时更新记录的一系列table。

rocketmq有两种方式来出发路由删除:
1)nameserver定时扫描brokerLiveTable,如果BrokerLive的lastUpdateTimestamp的时间戳距离当前时间超过120s,则认为broker失效,移除该broker.
2)broker正常关闭时,执行unregisterBroker指令。主动触发路由删除。

两种方式删除的逻辑都是一样的。

定时删除路由信息

前面我们知道在nameserver启动时,启动了两个定时任务,其中一个就是每隔10s定时扫描brokerLiveTable,移除不活跃broker。

    public boolean initialize() {
     
        //加载KV配置
        this.kvConfigManager.load();
        //创建NettyServer网络处理对象
        this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);
        //创建处理网络连接的线程池
        this.remotingExecutor =
            Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(), new ThreadFactoryImpl("RemotingExecutorThread_"));
        //注册业务处理器
        this.registerProcessor();
        //开启删除不活跃broker的定时任务,间隔10s扫描一次,移除不活跃broker
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
     

            @Override
            public void run() {
     
                NamesrvController.this.routeInfoManager.scanNotActiveBroker();
            }
        }, 5, 10, TimeUnit.SECONDS);
        //启动一个定时任务,每十分钟打印一次KV配置
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
     

            @Override
            public void run() {
     
                NamesrvController.this.kvConfigManager.printAllPeriodically();
            }
        }, 1, 10, TimeUnit.MINUTES);

scanNotActiveBroker方法

由上面看出,调用routeInfoManager.scanNotActiveBroker()方法进行路由删除

    public void scanNotActiveBroker() {
     
        //遍历brokerLiveTable路由表
        Iterator<Entry<String, BrokerLiveInfo>> it = this.brokerLiveTable.entrySet().iterator();
        while (it.hasNext()) {
     
            Entry<String, BrokerLiveInfo> next = it.next();
            //获取lastUpdateTimestamp
            long last = next.getValue().getLastUpdateTimestamp();
            //判断如果lastUpdateTimestamp的时间加上120,并且小于当前系统时间
            if ((last + BROKER_CHANNEL_EXPIRED_TIME) < System.currentTimeMillis()) {
     
                //关闭Channel
                RemotingUtil.closeChannel(next.getValue().getChannel());
                //移除此broker
                it.remove();
                log.warn("The broker channel expired, {} {}ms", next.getKey(), BROKER_CHANNEL_EXPIRED_TIME);
                //删除与该broker相关的路由信息
                this.onChannelDestroy(next.getKey(), next.getValue().getChannel());
            }
        }
    }

onChannelDestroy方法逻辑如下

    public void onChannelDestroy(String remoteAddr, Channel channel) {
     
        String brokerAddrFound = null;
        if (channel != null) {
     
            try {
     
                try {
     
                    //获取读锁
                    this.lock.readLock().lockInterruptibly();
                    Iterator<Entry<String, BrokerLiveInfo>> itBrokerLiveTable =
                        this.brokerLiveTable.entrySet().iterator();
                    while (itBrokerLiveTable.hasNext()) {
     
                        Entry<String, BrokerLiveInfo> entry = itBrokerLiveTable.next();
                        //根据channel,找到对应的brokerAddr
                        if (entry.getValue().getChannel() == channel) {
     
                            brokerAddrFound = entry.getKey();
                            break;
                        }
                    }
                } finally {
     
                    //释放读锁
                    this.lock.readLock().unlock();
                }
            } catch (Exception e) {
     
                log.error("onChannelDestroy Exception", e);
            }
        }

        if (null == brokerAddrFound) {
     
            brokerAddrFound = remoteAddr;
        } else {
     
            log.info("the broker's channel destroyed, {}, clean it's data structure at once", brokerAddrFound);
        }

        if (brokerAddrFound != null && brokerAddrFound.length() > 0) {
     
            try {
     
                try {
     
                    //申请写锁,根据brokerAddrFound从brokerLiveTable、filterServerTable中移除
                    this.lock.writeLock().lockInterruptibly();
                    this.brokerLiveTable.remove(brokerAddrFound);
                    this.filterServerTable.remove(brokerAddrFound);
                    String brokerNameFound = null;
                    boolean removeBrokerName = false;
                    //遍历brokerAddrTable
                    Iterator<Entry<String, BrokerData>> itBrokerAddrTable =
                        this.brokerAddrTable.entrySet().iterator();
                    while (itBrokerAddrTable.hasNext() && (null == brokerNameFound)) {
     
                        //获取BrokerData
                        BrokerData brokerData = itBrokerAddrTable.next().getValue();
                        //从BrokerData中获取BrokerAddrs
                        Iterator<Entry<Long, String>> it = brokerData.getBrokerAddrs().entrySet().iterator();
                        while (it.hasNext()) {
     //遍历BrokerAddrs
                            Entry<Long, String> entry = it.next();
                            Long brokerId = entry.getKey();
                            //拿到每一个brokerAddr
                            String brokerAddr = entry.getValue();
                            if (brokerAddr.equals(brokerAddrFound)) {
     
                                //找到具体的broker,从BrokerData中移除
                                brokerNameFound = brokerData.getBrokerName();
                                it.remove();
                                log.info("remove brokerAddr[{}, {}] from brokerAddrTable, because channel destroyed",
                                    brokerId, brokerAddr);
                                break;
                            }
                        }
                        //如果移除后,在brokerData中不再包含其他broker,则在brokerAddressTable中
                        //移除该brokerName对应的条目
                        if (brokerData.getBrokerAddrs().isEmpty()) {
     
                            removeBrokerName = true;
                            itBrokerAddrTable.remove();
                            log.info("remove brokerName[{}] from brokerAddrTable, because channel destroyed",
                                brokerData.getBrokerName());
                        }
                    }

                    if (brokerNameFound != null && removeBrokerName) {
     
                        //从clusterAddrTable中找到并从集群中移除。
                        Iterator<Entry<String, Set<String>>> it = this.clusterAddrTable.entrySet().iterator();
                        while (it.hasNext()) {
     
                            Entry<String, Set<String>> entry = it.next();
                            String clusterName = entry.getKey();
                            Set<String> brokerNames = entry.getValue();
                            boolean removed = brokerNames.remove(brokerNameFound);
                            if (removed) {
     
                                log.info("remove brokerName[{}], clusterName[{}] from clusterAddrTable, because channel destroyed",
                                    brokerNameFound, clusterName);
                                //移除后,集群中不包含任何broker,则将该集群从clusterAddrTable中移除
                                if (brokerNames.isEmpty()) {
     
                                    log.info("remove the clusterName[{}] from clusterAddrTable, because channel destroyed and no broker in this cluster",
                                        clusterName);
                                    it.remove();
                                }

                                break;
                            }
                        }
                    }

                    if (removeBrokerName) {
     
                        //遍历所有主题的队列
                        Iterator<Entry<String, List<QueueData>>> itTopicQueueTable =
                            this.topicQueueTable.entrySet().iterator();
                        while (itTopicQueueTable.hasNext()) {
     
                            Entry<String, List<QueueData>> entry = itTopicQueueTable.next();
                            String topic = entry.getKey();
                            List<QueueData> queueDataList = entry.getValue();

                            Iterator<QueueData> itQueueData = queueDataList.iterator();
                            while (itQueueData.hasNext()) {
     
                                QueueData queueData = itQueueData.next();
                                //如果队列中包含了当前broker的队列,则移除
                                if (queueData.getBrokerName().equals(brokerNameFound)) {
     
                                    itQueueData.remove();
                                    log.info("remove topic[{} {}], from topicQueueTable, because channel destroyed",
                                        topic, queueData);
                                }
                            }
                            //如果topic只包含待移除broker的队列的话,从路由表中删除该topic
                            if (queueDataList.isEmpty()) {
     
                                itTopicQueueTable.remove();
                                log.info("remove topic[{}] all queue, from topicQueueTable, because channel destroyed",
                                    topic);
                            }
                        }
                    }
                } finally {
     
                    //释放锁,完成路由删除
                    this.lock.writeLock().unlock();
                }
            } catch (Exception e) {
     
                log.error("onChannelDestroy Exception", e);
            }
        }
    }

8、路由发现

当topic路由发生变化,nameserver不会主动统治客户端,而是由客户端定时拉取主题最新的路由信息。

DefaultRequestProcessor类中根据topic获取路由信息的方法getRouteInfoByTopic

    public RemotingCommand getRouteInfoByTopic(ChannelHandlerContext ctx,
        RemotingCommand request) throws RemotingCommandException {
     
        final RemotingCommand response = RemotingCommand.createResponseCommand(null);
        final GetRouteInfoRequestHeader requestHeader =
            (GetRouteInfoRequestHeader) request.decodeCommandCustomHeader(GetRouteInfoRequestHeader.class);
        /*
         调用RouteInfoManager方法,从路由表topicQueueTable、brokerAddrTable、
         filterServerTable中分别填充TopicRouteData中的List、List
         和filterServerTable地址表
         */
        TopicRouteData topicRouteData = this.namesrvController.getRouteInfoManager().pickupTopicRouteData(requestHeader.getTopic());

        if (topicRouteData != null) {
     //获取到对应的路由信息
            //如果该主题为顺序消息
            if (this.namesrvController.getNamesrvConfig().isOrderMessageEnable()) {
     
                //从NameServerKVconfig中获取关于顺序消息相关的配置
                String orderTopicConf =
                    this.namesrvController.getKvConfigManager().getKVConfig(NamesrvUtil.NAMESPACE_ORDER_TOPIC_CONFIG,
                        requestHeader.getTopic());
                //填充路由信息
                topicRouteData.setOrderTopicConf(orderTopicConf);
            }

            byte[] content = topicRouteData.encode();
            response.setBody(content);
            response.setCode(ResponseCode.SUCCESS);
            response.setRemark(null);
            return response;
        }
        //如果没找到则使用TOPIC_NOT_EXIST,表示没有找到对应的路由
        response.setCode(ResponseCode.TOPIC_NOT_EXIST);
        response.setRemark("No topic route info in name server for the topic: " + requestHeader.getTopic()
            + FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL));
        return response;
    }

你可能感兴趣的:(RocketMQ系列,分布式,中间件)