Tx-lcn lcn-manager 服务端执行逻辑（5）

服务端很简单，只需要一个注解@EnableTransactionManagerServer 配置在启动类上就能开启分布式事务服务端功能。

一 、启动入口

注解@EnableTransactionManagerServer

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Import(value = {TMAutoConfiguration.class})
public @interface EnableTransactionManagerServer {
}

引入了TMAutoConfiguration类，所有的功能都在TMAutoConfiguration类中，从名称上看是自动配置类

@Configuration
@ComponentScan
@Import({TxLoggerConfiguration.class, MessageConfiguration.class})
@EnableJpaRepositories("com.codingapi.txlcn.tm.support.db.jpa")
@EntityScan("com.codingapi.txlcn.tm.support.db.domain")
public class TMAutoConfiguration {

    @Bean(destroyMethod = "shutdown")
    public ExecutorService executorService() {
        int coreSize = Runtime.getRuntime().availableProcessors() * 2;
        return new ThreadPoolExecutor(coreSize, coreSize, 0L, TimeUnit.MILLISECONDS,
                new LinkedBlockingQueue<>()) {
            @Override
            public void shutdown() {
                super.shutdown();
                try {
                    this.awaitTermination(10, TimeUnit.MINUTES);
                } catch (InterruptedException ignored) {
                }
            }
        };
    }

    @Bean
    public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder) {
        return restTemplateBuilder.build();
    }

    @Bean
    @ConditionalOnMissingBean
    public FastStorageProvider fastStorageProvider(RedisTemplate redisTemplate,
                                                   StringRedisTemplate stringRedisTemplate, TxManagerConfig managerConfig) {
        return () -> new RedisStorage(redisTemplate, stringRedisTemplate, managerConfig);
    }

    @Bean
    public FastStorage fastStorage(FastStorageProvider fastStorageProvider) {
        return fastStorageProvider.provide();
    }

    @Bean
    public TxLcnApplicationRunner txLcnApplicationRunner(ApplicationContext applicationContext) {
        return new TxLcnApplicationRunner(applicationContext);
    }

    @Bean
    @ConditionalOnMissingBean
    public ModIdProvider modIdProvider(ConfigurableEnvironment environment, ServerProperties serverProperties) {
        return () -> ApplicationInformation.modId(environment, serverProperties);
    }
}

总结：这里算是所有的配置

1、引入日志配置TxLoggerConfiguration （日志就不细说了）

2、引入消息配置MessageConfiguration类，配置了一些客户端与服务端通信的类，心跳监听HeartbeatListener

3、开启了jpa支持把一些事务信息写入数据库 EnableJpaRepositories

4、自定义线程池

5、构建了restTemplate   Bean

6、构建FastStorageProvider  Bean 此类用来提供快速存储类，需要三个参数，这里默认构建RedisStorage

7、根据FastStorageProvider 构建RedisStorage

8、构建TxLcnApplicationRunner  Bean 用于spring boot启动后做些事情。（很重要）

9、构建ModIdProvider Bean 返回模块标识。
二、MessageConfiguration 配置类

@Configuration
@ComponentScan
@Slf4j
@Data
public class MessageConfiguration {


    @Bean
    @ConditionalOnMissingBean
    @ConfigurationProperties("tx-lcn.message.netty")
    //rpc的配置如心跳，重试，缓存锁数量
    public RpcConfig rpcConfig() {
        return new RpcConfig();
    }

    @Bean
    @ConditionalOnMissingBean
    //rpc应答类，只是打印了消息
    public RpcAnswer rpcClientAnswer() {
        return rpcCmd -> log.info("cmd->{}", rpcCmd);
    }

    @Bean
    @ConditionalOnMissingBean
    //rpc负载均衡策略，采用RandomLoadBalance类
    public RpcLoadBalance rpcLoadBalance() {
        return new RandomLoadBalance();
    }


    @Bean
    @ConditionalOnMissingBean
    //客户端初始化回调类，只打印日志
    public ClientInitCallBack clientInitCallBack() {
        return new DefaultClientInitCallback();
    }

    @Bean
    @ConditionalOnMissingBean
    //rpc连接监听器，默认实现
    public RpcConnectionListener rpcConnectionListener(){
        return new DefaultRpcConnectionListener();
    }

    @Bean
    @ConditionalOnMissingBean
    //心跳监听器，默认空实现
    public HeartbeatListener heartbeatListener(){
        return new DefaultHeartbeatListener();
    }
}

RedisStorage 快速存储类

FastStorage的唯一实现，用于操作redis完成事务的操作，内部封装了RedisTemplate与StringRedisTemplate。

其功能比如：事务组相关操作；事务状态相关操作；分布式锁的操作；TM的机器列表操作；token操作等。

（代码可忽略，了解类的作用即可）

@Slf4j
@Data
public class RedisStorage implements FastStorage {

    private static final String REDIS_GROUP_PREFIX = "tm:group:";

    private static final String REDIS_GROUP_STATE = REDIS_GROUP_PREFIX + "transactionState:";

    private static final String REDIS_TOKEN_PREFIX = "tm.token";

    private static final String REDIS_TM_LIST = "tm.instances";

    private static final String REDIS_MACHINE_ID_MAP_PREFIX = "tm.machine.id.gen:";

    private RedisTemplate redisTemplate;

    private StringRedisTemplate stringRedisTemplate;

    private TxManagerConfig managerConfig;

    public RedisStorage() {
    }

    public RedisStorage(RedisTemplate redisTemplate, StringRedisTemplate stringRedisTemplate,
                        TxManagerConfig managerConfig) {
        this.redisTemplate = redisTemplate;
        this.managerConfig = managerConfig;
        this.stringRedisTemplate = stringRedisTemplate;
    }

    @Override
    public void initGroup(String groupId) {
        redisTemplate.opsForHash().put(REDIS_GROUP_PREFIX + groupId, "root", "");
        redisTemplate.expire(REDIS_GROUP_PREFIX + groupId, managerConfig.getDtxTime() + 10000, TimeUnit.MILLISECONDS);
    }

    @Override
    public boolean containsGroup(String groupId) {
        return Optional.ofNullable(redisTemplate.hasKey(REDIS_GROUP_PREFIX + groupId)).orElse(false);
    }

    @Override
    public List findTransactionUnitsFromGroup(String groupId) throws FastStorageException {
        Map units = redisTemplate.opsForHash().entries(REDIS_GROUP_PREFIX + groupId);
        return units.entrySet().stream()
                .filter(objectObjectEntry -> !objectObjectEntry.getKey().equals("root"))
                .map(objectObjectEntry -> (TransactionUnit) objectObjectEntry.getValue()).collect(Collectors.toList());
    }

    @Override
    public void saveTransactionUnitToGroup(String groupId, TransactionUnit transactionUnit) throws FastStorageException {
        if (Optional.ofNullable(redisTemplate.hasKey(REDIS_GROUP_PREFIX + groupId)).orElse(false)) {
            redisTemplate.opsForHash().put(REDIS_GROUP_PREFIX + groupId, transactionUnit.getUnitId(), transactionUnit);
            return;
        }
        throw new FastStorageException("attempts to the non-existent transaction group " + groupId,
                FastStorageException.EX_CODE_NON_GROUP);
    }

    @Override
    public void clearGroup(String groupId) {
        log.debug("remove group:{} from redis.", groupId);
        redisTemplate.delete(REDIS_GROUP_PREFIX + groupId);
    }

    @Override
    public void saveTransactionState(String groupId, int state) throws FastStorageException {
        redisTemplate.opsForValue().set(REDIS_GROUP_STATE + groupId, String.valueOf(state));
        redisTemplate.expire(REDIS_GROUP_STATE + groupId, managerConfig.getDtxTime() + 10000, TimeUnit.MILLISECONDS);
    }

    @Override
    public int getTransactionState(String groupId) throws FastStorageException {
        Object state = redisTemplate.opsForValue().get(REDIS_GROUP_STATE + groupId);
        if (Objects.isNull(state)) {
            return -1;
        }

        try {
            return Integer.valueOf(state.toString());
        } catch (Exception e) {
            return -1;
        }
    }

    @Override
    public void acquireLocks(String contextId, Set locks, LockValue lockValue) throws FastStorageException {
        // 未申请锁则为申请正常
        if (Objects.isNull(locks) || locks.isEmpty()) {
            return;
        }
        Map lockIds = locks.stream().collect(Collectors.toMap(lock -> contextId + lock, lock -> lockValue));
        String firstLockId = contextId + new ArrayList<>(locks).get(0);
        Boolean result = redisTemplate.opsForValue().multiSetIfAbsent(lockIds);
        if (!Optional.ofNullable(result).orElse(false)) {
            LockValue hasLockValue = (LockValue) redisTemplate.opsForValue().get(firstLockId);
            if (Objects.isNull(hasLockValue)) {
                throw new FastStorageException("acquire locks fail.", FastStorageException.EX_CODE_REPEAT_LOCK);
            }
            // 不在同一个DTX下，已存在的锁是排它锁 或者 新请求的不是共享锁时， 获取锁失败
            if (Objects.isNull(lockValue.getGroupId()) || !lockValue.getGroupId().equals(hasLockValue.getGroupId())) {
                if (hasLockValue.getLockType() == DTXLocks.X_LOCK || lockValue.getLockType() != DTXLocks.S_LOCK) {
                    throw new FastStorageException("acquire locks fail.", FastStorageException.EX_CODE_REPEAT_LOCK);
                }
            }
            redisTemplate.opsForValue().multiSet(lockIds);
        }

        // 锁超时时间设置
        lockIds.forEach((k, v) -> redisTemplate.expire(k, managerConfig.getDtxTime(), TimeUnit.MILLISECONDS));
    }

    @Override
    public void releaseLocks(String cate, Set locks) {
        redisTemplate.delete(locks.stream().map(lock -> (cate + lock)).collect(Collectors.toSet()));
    }

    @Override
    public void saveToken(String token) {
        Objects.requireNonNull(token);
        redisTemplate.opsForList().leftPush(REDIS_TOKEN_PREFIX, token);
        redisTemplate.expire(REDIS_TOKEN_PREFIX, 20, TimeUnit.MINUTES);

        Long size = redisTemplate.opsForList().size(REDIS_TOKEN_PREFIX);
        if (Objects.nonNull(size) && size > 3) {
            redisTemplate.opsForList().rightPop(REDIS_TOKEN_PREFIX);
        }
    }

    @Override
    public List findTokens() {
        Long size = redisTemplate.opsForList().size(REDIS_TOKEN_PREFIX);
        if (Objects.isNull(size)) {
            return Collections.emptyList();
        }
        return Objects.requireNonNull(redisTemplate.opsForList().range(REDIS_TOKEN_PREFIX, 0, size))
                .stream()
                .map(Object::toString).collect(Collectors.toList());
    }

    @Override
    public void removeToken(String token) {
        redisTemplate.delete(REDIS_TOKEN_PREFIX);
    }

    @Override
    public void saveTMProperties(TMProperties tmProperties) {
        Objects.requireNonNull(tmProperties);
        stringRedisTemplate.opsForHash().put(REDIS_TM_LIST,
                tmProperties.getHost() + ":" + tmProperties.getTransactionPort(), String.valueOf(tmProperties.getHttpPort()));
    }

    @Override
    public List findTMProperties() {
        return stringRedisTemplate.opsForHash().entries(REDIS_TM_LIST).entrySet().stream()
                .map(entry -> {
                    String[] args = ApplicationInformation.splitAddress(entry.getKey().toString());
                    TMProperties tmProperties = new TMProperties();
                    tmProperties.setHost(args[0]);
                    tmProperties.setTransactionPort(Integer.valueOf(args[1]));
                    tmProperties.setHttpPort(Integer.parseInt(entry.getValue().toString()));
                    return tmProperties;
                }).collect(Collectors.toList());
    }

    @Override
    public void removeTMProperties(String host, int transactionPort) {
        Objects.requireNonNull(host);
        redisTemplate.opsForHash().delete(REDIS_TM_LIST, host + ":" + transactionPort);
        log.debug("removed TM {}:{}", host, transactionPort);
    }

    private static final String GLOBAL_CONTEXT = "root";
    private static final String GLOBAL_LOCK_ID = "global.lock";

    private void acquireGlobalXLock() {
        LockValue lockValue = new LockValue();
        lockValue.setLockType(DTXLocks.X_LOCK);
        while (true) {
            try {
                acquireLocks(GLOBAL_CONTEXT, Sets.newHashSet(GLOBAL_LOCK_ID), lockValue);
                break;
            } catch (FastStorageException ignored) {
            }
        }
    }

    private void releaseGlobalXLock() {
        releaseLocks(GLOBAL_CONTEXT, Sets.newHashSet(GLOBAL_LOCK_ID));
    }

    @Override
    public long acquireMachineId(long machineMaxSize, long timeout) throws FastStorageException {
        try {
            acquireGlobalXLock();
            stringRedisTemplate.opsForValue().setIfAbsent(REDIS_MACHINE_ID_MAP_PREFIX + "cur_id", "-1");
            for (int i = 0; i < machineMaxSize; i++) {
                long curId = Objects.requireNonNull(
                        stringRedisTemplate.opsForValue().increment(REDIS_MACHINE_ID_MAP_PREFIX + "cur_id", 1));
                if (curId > machineMaxSize) {
                    stringRedisTemplate.opsForValue().set(REDIS_MACHINE_ID_MAP_PREFIX + "cur_id", "0");
                    curId = 0;
                }
                if (Optional
                        .ofNullable(stringRedisTemplate.hasKey(REDIS_MACHINE_ID_MAP_PREFIX + curId))
                        .orElse(true)) {
                    continue;
                }
                stringRedisTemplate.opsForValue().set(REDIS_MACHINE_ID_MAP_PREFIX + curId, "", timeout, TimeUnit.MILLISECONDS);
                return curId;
            }
            throw new FastStorageException("non can used machine id", FastStorageException.EX_CODE_NON_MACHINE_ID);
        } finally {
            releaseGlobalXLock();
        }
    }

    @Override
    public void refreshMachines(long timeout, long... machines) {
        try {
            stringRedisTemplate.setEnableTransactionSupport(true);
            stringRedisTemplate.multi();
            for (long mac : machines) {
                stringRedisTemplate.opsForValue().set(REDIS_MACHINE_ID_MAP_PREFIX + mac, "", timeout, TimeUnit.MILLISECONDS);
            }
            stringRedisTemplate.exec();
        } catch (Throwable e) {
            stringRedisTemplate.discard();
        } finally {
            stringRedisTemplate.setEnableTransactionSupport(false);
        }
    }
}

TxLcnApplicationRunner  框架启动运行类（比较重要）

@Bean
    public TxLcnApplicationRunner txLcnApplicationRunner(ApplicationContext applicationContext) {
        // 通过注入Spring 的上下文applicationContext 来构建TxLcnApplicationRunner对象
        return new TxLcnApplicationRunner(applicationContext);
    }

TxLcnApplicationRunner类实现了ApplicationRunner接口，并实现了run方法，Spring boot在启动后会自动调用实现了ApplicationRunner接口类的run方法，代码如下

public class TxLcnApplicationRunner implements ApplicationRunner, DisposableBean {
    
    private final ApplicationContext applicationContext;
    
    private List initializers;
    
    @Autowired
    public TxLcnApplicationRunner(ApplicationContext applicationContext) {
        this.applicationContext = applicationContext;
    }
    
    @Override
    public void run(ApplicationArguments args) throws Exception {
        Map runnerMap = applicationContext.getBeansOfType(TxLcnInitializer.class);
        
        //获取容器中实现了TxLcnInitializer接口的类，并排序
        initializers = runnerMap.values().stream().sorted(Comparator.comparing(TxLcnInitializer::order))
                .collect(Collectors.toList());
        
        //循环调用txLcnInitializer的init（）方法 对每个txLcnInitializer 进行初始化init()
        for (TxLcnInitializer txLcnInitializer : initializers) {
            txLcnInitializer.init();
        }
    }
    
    @Override
    public void destroy() throws Exception {
        for (TxLcnInitializer txLcnInitializer : initializers) {
            txLcnInitializer.destroy();
        }
    }
    
}

重点是run方法，找到容器中的所有的TxLcnInitializer并调用其init方法。

重要的工作在于执行所有TxLcnInitializer的init方法。

三、TxLcnInitializer初始化之TMAutoCluster

TMAutoCluster从名称上看是自动集群，当我们启动一个新的服务端时，不用额外的配置也不需要重新启动客户端。所有的客户端都会感知到新的服务端并且与之链接。

整体的逻辑是这样的，如果启动一个服务端A，启动时这个A会把自己的信息存入redis的hash中，hash为tm.instances,hk为host:TransactionPort，hv为HttpPort。

例：

如果服务端A host 为192.168.120.10 设置的server.port=7970,则redis 中的tm.instances 一个值为KEY 为192.168.120.10：7970,VALUE8070。然后客户端都连接这个服务端A。

过段时间后，又需要启动一个服务端B，启动B时会先去（A服务也会这样）redis上获取tm.instances上所有的值（这里只有服务端A），排除掉自己的信息后（如果有），根据redis存储的信息通过restTemplate以自身的地址信息为参数去调用服务端A，服务端A收到服务端B的请求会给与自己相连的所有的channel发送信息要求所有的客户端连接服务端B，客户端接收到服务端A的消息后会新启动netty客户端去连接服务端B。这就是自动集群的所有流程。
 

@Override
    public void init() throws Exception {

        // 1. 通知 TC 建立连接
        List tmList = fastStorage.findTMProperties().stream()
                .filter(tmProperties ->
                        !tmProperties.getHost().equals(txManagerConfig.getHost()) || !tmProperties.getTransactionPort().equals(txManagerConfig.getPort()))
                .collect(Collectors.toList());
        for (TMProperties properties : tmList) {
            NotifyConnectParams notifyConnectParams = new NotifyConnectParams();
            notifyConnectParams.setHost(txManagerConfig.getHost());
            notifyConnectParams.setPort(txManagerConfig.getPort());
            String url = String.format(MANAGER_REFRESH_URL, properties.getHost(), properties.getHttpPort());
            try {
                //调用其他服务
                ResponseEntity res = restTemplate.postForEntity(url, notifyConnectParams, Boolean.class);
                if (res.getStatusCode().equals(HttpStatus.OK) || res.getStatusCode().is5xxServerError()) {
                    log.info("manager auto refresh res->{}", res);
                    break;
                } else {
                    fastStorage.removeTMProperties(properties.getHost(), properties.getTransactionPort());
                }
            } catch (Exception e) {
                log.error("manager auto refresh error: {}", e.getMessage());
                //check exception then remove.
                if (e instanceof ResourceAccessException) {
                    ResourceAccessException resourceAccessException = (ResourceAccessException) e;
                    if (resourceAccessException.getCause() != null && resourceAccessException.getCause() instanceof ConnectException) {
                        //can't access .
                        fastStorage.removeTMProperties(properties.getHost(), properties.getTransactionPort());
                    }
                }
            }
        }

        // 2. 保存TM 到快速存储
        if (StringUtils.hasText(txManagerConfig.getHost())) {
            TMProperties tmProperties = new TMProperties();
            tmProperties.setHttpPort(ApplicationInformation.serverPort(serverProperties));
            tmProperties.setHost(txManagerConfig.getHost());
            tmProperties.setTransactionPort(txManagerConfig.getPort());
            fastStorage.saveTMProperties(tmProperties);
        }
    }

主要做了两件事

1、获取redis的值，排除自己的信息后用restTemplate去调用所有redis存储的信息地址。

2、把自己的信息保存到redis

public class TxManagerController {
 
    @Autowired
    private ManagerService managerService;
 
    @PostMapping("/refresh")
    public boolean refresh(@RequestBody NotifyConnectParams notifyConnectParams) throws RpcException {
        return managerService.refresh(notifyConnectParams);
    }
}

 @Override
    public boolean refresh(NotifyConnectParams notifyConnectParams) throws RpcException {
        List keys = rpcClient.loadAllRemoteKey();
        if (keys != null && keys.size() > 0) {
            for (String key : keys) {
                rpcClient.send(key, MessageCreator.newTxManager(notifyConnectParams));
            }
        }
        return true;
    }

public List loadAllRemoteKey() {
        List allKeys = new ArrayList<>();
        for (Channel channel : channels) {
            allKeys.add(channel.remoteAddress().toString());
        }
        return allKeys;
    }

可以看到调用是对每一个连接的channel都会去通知

保存TM信息

public void saveTMProperties(TMProperties tmProperties) {
        Objects.requireNonNull(tmProperties);
        stringRedisTemplate.opsForHash().put(REDIS_TM_LIST,
                tmProperties.getHost() + ":" + tmProperties.getTransactionPort(), String.valueOf(tmProperties.getHttpPort()));
    }