RocketMQ实现可靠消息最终一致性

RocketMQ实现可靠消息最终一致性

业务说明

本实例通过RocketMQ中间件实现可靠消息最终一致性分布式事务，模拟两个账户的转账交易过程。
两个账户在分别在不同的银行(张三在bank1、李四在bank2)，bank1、bank2是两个微服务。
交易过程是，张三 给李四转账指定金额。

上述交易步骤，张三扣减金额与给bank2发转账消息，两个操作必须是一个整体性的事务。

程序组成部分

数据库：MySQL-8.X （包括bank1和bank2 两个数据库）
JDK:1.8.X
微服务框架：SpringBoot2.3、SpringCloudHoxton
RocketMQ：rocketmq-spring-boot-starter.2.0.4
微服务及数据库的关系：
dtx/rocker-bank-mq/bank1-server-mq 银行1 操作张三用户，连接bank1
dtx/rocker-bank-mq/bank2-server-mq 银行2 操作李四用户，连接bank2

创建数据库

bank1库，包含张三的账户

DROP DATABASE IF  EXISTS  `bank1`;
CREATE DATABASE `bank1` CHARACTER SET 'utf8mb4';

USE `bank1`;

DROP TABLE IF EXISTS `account_info`;
CREATE TABLE `account_info` (
		id bigint ( 20 ) NOT NULL  AUTO_INCREMENT COMMENT '主键',
		account_name VARCHAR ( 100 ) COMMENT '账户姓名',
		account_no VARCHAR ( 100 ) COMMENT '账户卡号',
		account_password VARCHAR ( 100 ) COMMENT '账户密码',
		account_balance DECIMAL ( 10,2) COMMENT '账户余额',
		PRIMARY KEY ( id ) 
) COMMENT = '账户表';

INSERT INTO `account_info` VALUES ('1' , '张三的账户','1', '', 10000 );

bank2库，包含李四的账户

DROP DATABASE IF  EXISTS  `bank2`;
CREATE DATABASE `bank2` CHARACTER SET 'utf8mb4';

USE `bank2`;

DROP TABLE IF EXISTS `account_info`;
CREATE TABLE `account_info` (
		id bigint ( 20 ) NOT NULL  AUTO_INCREMENT COMMENT '主键',
		account_name VARCHAR ( 100 ) COMMENT '账户姓名',
		account_no VARCHAR ( 100 ) COMMENT '账户卡号',
		account_password VARCHAR ( 100 ) COMMENT '账户密码',
		account_balance DECIMAL ( 10,2)  COMMENT '账户余额',
		PRIMARY KEY ( id ) 
) COMMENT = '账户表';

INSERT INTO `account_info` VALUES ('1' , '李四的账户','2', '', 0);

每个数据库都创建 de_duplication

DROP TABLE IF EXISTS `de_duplication`;
CREATE TABLE `de_duplication` (
		tx_no VARCHAR ( 64) NOT NULL COMMENT '事务ID',
		create_time datetime COMMENT '创建时间',
		PRIMARY KEY ( tx_no ) 
) COMMENT = '事务执行记录表';

搭建项目

启动RocketMQ

（1）下载RocketMQ服务器
下载地址：http://mirrors.tuna.tsinghua.edu.cn/apache/rocketmq/4.5.0/rocketmq-all-4.5.0-bin- release.zip
（2）解压并启动
启动nameserver:

set ROCKETMQ_HOME=[rocketmq服务端解压路径] start [rocketmq服务端解压路径]/bin/mqnamesrv.cmd

启动broker:

set ROCKETMQ_HOME=[rocketmq服务端解压路径] start [rocketmq服务端解压路径]/bin/mqbroker.cmd ‐n 127.0.0.1:9876 autoCreateTopicEnable=true

**我使用的是docker 容器创建的 rockerMQ **

docker-compose.yml

version: '3.5'
services:
  rmqnamesrv:
    image: foxiswho/rocketmq:server
    container_name: rmqnamesrv
    ports:
      - 9876:9876
    volumes:
      - ./data/logs:/opt/logs
      - ./data/store:/opt/store
    environment:
        JAVA_OPTS: " -Duser.home=/opt"
        JAVA_OPT_EXT: "-server -Xms128m -Xmx128m -Xmn128m"
    networks:
        rmq:
          aliases:
            - rmqnamesrv

  rmqbroker:
    image: foxiswho/rocketmq:broker
    container_name: rmqbroker
    ports:
      - 10909:10909
      - 10911:10911
    volumes:
      - ./data/logs:/opt/logs
      - ./data/store:/opt/store
      - ./data/brokerconf/broker.conf:/etc/rocketmq/broker.conf
    environment:
        NAMESRV_ADDR: "rmqnamesrv:9876"
        JAVA_OPTS: " -Duser.home=/opt"
        JAVA_OPT_EXT: "-server -Xms128m -Xmx128m -Xmn128m"
    command: mqbroker -c /etc/rocketmq/broker.conf
    depends_on:
      - rmqnamesrv
    networks:
      rmq:
        aliases:
          - rmqbroker

  rmqconsole:
    image: styletang/rocketmq-console-ng
    container_name: rmqconsole
    ports:
      - 8080:8080
    environment:
        JAVA_OPTS: "-Drocketmq.namesrv.addr=rmqnamesrv:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false"
    depends_on:
      - rmqnamesrv
    networks:
      rmq:
        aliases:
          - rmqconsole

networks:
  rmq:
    name: rmq
    driver: bridge

broker.conf
RocketMQ Broker 需要一个配置文件，按照上面的 Compose 配置，我们需要在 ./data/brokerconf/ 目录下创建一个名为 broker.conf 的配置文件，内容如下：

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.


# 所属集群名字
brokerClusterName=DefaultCluster

# broker 名字，注意此处不同的配置文件填写的不一样，如果在 broker-a.properties 使用: broker-a,
# 在 broker-b.properties 使用: broker-b
brokerName=broker-a

# 0 表示 Master，> 0 表示 Slave
brokerId=0

# nameServer地址，分号分割
# namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876

# 启动IP,如果 docker 报 com.alibaba.rocketmq.remoting.exception.RemotingConnectException: connect to <192.168.0.120:10909> failed
# 解决方式1 加上一句 producer.setVipChannelEnabled(false);，解决方式2 brokerIP1 设置宿主机IP，不要使用docker 内部IP
# brokerIP1=192.168.0.253

# 在发送消息时，自动创建服务器不存在的topic，默认创建的队列数
defaultTopicQueueNums=4

# 是否允许 Broker 自动创建 Topic，建议线下开启，线上关闭 ！！！这里仔细看是 false，false，false
autoCreateTopicEnable=true

# 是否允许 Broker 自动创建订阅组，建议线下开启，线上关闭
autoCreateSubscriptionGroup=true

# Broker 对外服务的监听端口
listenPort=10911

# 删除文件时间点，默认凌晨4点
deleteWhen=04

# 文件保留时间，默认48小时
fileReservedTime=120

# commitLog 每个文件的大小默认1G
mapedFileSizeCommitLog=1073741824

# ConsumeQueue 每个文件默认存 30W 条，根据业务情况调整
mapedFileSizeConsumeQueue=300000

# destroyMapedFileIntervalForcibly=120000
# redeleteHangedFileInterval=120000
# 检测物理文件磁盘空间
diskMaxUsedSpaceRatio=88
# 存储路径
# storePathRootDir=/home/ztztdata/rocketmq-all-4.1.0-incubating/store
# commitLog 存储路径
# storePathCommitLog=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/commitlog
# 消费队列存储
# storePathConsumeQueue=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/consumequeue
# 消息索引存储路径
# storePathIndex=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/index
# checkpoint 文件存储路径
# storeCheckpoint=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/checkpoint
# abort 文件存储路径
# abortFile=/home/ztztdata/rocketmq-all-4.1.0-incubating/store/abort
# 限制的消息大小
maxMessageSize=65536

# flushCommitLogLeastPages=4
# flushConsumeQueueLeastPages=2
# flushCommitLogThoroughInterval=10000
# flushConsumeQueueThoroughInterval=60000

# Broker 的角色
# - ASYNC_MASTER 异步复制Master
# - SYNC_MASTER 同步双写Master
# - SLAVE
brokerRole=ASYNC_MASTER

# 刷盘方式
# - ASYNC_FLUSH 异步刷盘
# - SYNC_FLUSH 同步刷盘
flushDiskType=ASYNC_FLUSH

# 发消息线程池数量
# sendMessageThreadPoolNums=128
# 拉消息线程池数量
# pullMessageThreadPoolNums=128

RocketMQ 控制台
访问 http://rmqIP:8080 登入控制台

搭建项目

1. pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0modelVersion>
    <parent>
        <groupId>org.springframework.bootgroupId>
        <artifactId>spring-boot-starter-parentartifactId>
        <version>2.3.1.RELEASEversion>
        <relativePath/> 
    parent>
    <groupId>com.wry.dtxgroupId>
    <artifactId>bank1-server-mqartifactId>
    <version>0.0.1-SNAPSHOTversion>
    <name>bank1-server-mqname>
    <description>Demo project for Spring Bootdescription>

    <properties>
        <java.version>1.8java.version>
        <spring-cloud.version>Hoxton.SR5spring-cloud.version>
        <spring-cloud-alibaba.version>2.2.0.RELEASEspring-cloud-alibaba.version>
        <mybatis-plus.version>3.3.2mybatis-plus.version>
        <mysql.version>8.0.19mysql.version>
        <rocketmq.version>2.0.4rocketmq.version>
    properties>

    <dependencies>
        
        <dependency>
            <groupId>org.apache.rocketmqgroupId>
            <artifactId>rocketmq-spring-boot-starterartifactId>
            <version>${rocketmq.version}version>
        dependency>
        
        <dependency>
            <groupId>org.springframework.bootgroupId>
            <artifactId>spring-boot-starter-webartifactId>
        dependency>
        <dependency>
            <groupId>org.springframework.bootgroupId>
            <artifactId>spring-boot-starter-actuatorartifactId>
        dependency>
        <dependency>
            <groupId>org.springframework.bootgroupId>
            <artifactId>spring-boot-starter-testartifactId>
            <scope>testscope>
        dependency>
        <dependency>
            <groupId>mysqlgroupId>
            <artifactId>mysql-connector-javaartifactId>
            <version>${mysql.version}version>
        dependency>
        <dependency>
            <groupId>com.baomidougroupId>
            <artifactId>mybatis-plus-boot-starterartifactId>
            <version>${mybatis-plus.version}version>
        dependency>
    dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloudgroupId>
                <artifactId>spring-cloud-dependenciesartifactId>
                <version>${spring-cloud.version}version>
                <type>pomtype>
                <scope>importscope>
            dependency>
            <dependency>
                <groupId>com.alibaba.cloudgroupId>
                <artifactId>spring-cloud-alibaba-dependenciesartifactId>
                <version>${spring-cloud-alibaba.version}version>
                <type>pomtype>
                <scope>importscope>
            dependency>
        dependencies>
    dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.bootgroupId>
                <artifactId>spring-boot-maven-pluginartifactId>
            plugin>
        plugins>
    build>

project>

2. application.yml

server:
  port: 8005

spring:
  application:
    name: bank1-server-mq
  datasource:
    driver-class-name: com.mysql.cj.jdbc.Driver
    url: jdbc:mysql://39.96.3.100:3306/bank1?allowPublicKeyRetrieval=true&useUnicode=true&characterEncoding=utf-8&useSSL=false&serverTimezone=UTC&nullCatalogMeansCurrent=true
    username: root
    password: 123456

logging:
  level:
    root: info

mybatis-plus:
  mapper-locations: classpath:/mapper/*.xml
  typeAliasesPackage: com.wry.dtx.bank1.entity
  global-config:
    db-config:
      field-strategy: not-empty
      id-type: auto
      db-type: mysql

ribbon:
  ReadTimeout: 3000
  ConnectTimeout: 3000

rocketmq:
  name-server: 39.96.3.100:9876
  producer:
    group: producer_bank1

3. 启动类

package com.wry.dtx.bank1;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Bank1ServerMqApplication {

    public static void main(String[] args) {
        SpringApplication.run(Bank1ServerMqApplication.class, args);
    }

}

具体代码可参考Github:https://github.com/hobbyWang/DistributedTransaction/tree/master/rocket-bank-mq

小结

可靠消息最终一致性就是保证消息从生产方经过消息中间件传递到消费方的一致性，本案例使用了RocketMQ作为 消息中间件,
主要解决了两个功能：
1、本地事务与消息发送的原子性问题。
2、事务参与方接收消息的可靠性。

可靠消息最终一致性事务适合执行周期长且实时性要求不高的场景。引入消息机制后，同步的事务操作变为基于消 息执行的异步操作, 避免了分布式事务中的同步阻塞操作的影响，并实现了两个服务的解耦。