分布式中间件（二）：RocketMQ 应用

RocketMQ原生API使用

测试环境搭建

<dependency>
    <groupId>org.apache.rocketmqgroupId>
    <artifactId>rocketmq-clientartifactId>
    <version>4.9.1version>
dependency>
<dependency>
    <groupId>org.apache.rocketmqgroupId>
    <artifactId>rocketmq-aclartifactId>
    <version>4.9.1version>
dependency>

RocketMQ的编程模型

消息发送者的固定步骤

• 1.创建消息生产者producer，并制定生产者组名
• 2.指定Nameserver地址
• 3.启动producer
• 4.创建消息对象，指定主题Topic、Tag和消息体
• 5.发送消息
• 6.关闭生产者producer

消息消费者的固定步骤

• 1.创建消费者Consumer，制定消费者组名
• 2.指定Nameserver地址
• 3.订阅主题Topic和Tag
• 4.设置回调函数，处理消息
• 5.启动消费者consumer

RocketMQ的消息样例

一、基本样例

1、同步发送消息 SendResult sendResult = producer.send(msg);

2、异步发送消息 producer.send(msg, new SendCallback() {....});

3、单向发送消息 producer.sendOneway(msg);

4、使用消费者消费消息 DefaultLitePullConsumerImpl 和 DefaultMQPushConsumer

RocketMQ的推模式也是由拉模式封装出来的

二、顺序消息 RocketMQ保证的是消息的局部有序，而不是全局有序

SendResult sendResult = producer.send(msg, new MessageQueueSelector() {....}, orderId);

consumer.registerMessageListener(new MessageListenerOrderly() {...});

三、广播消息 consumer.setMessageModel(MessageModel.BROADCASTING);

四、延迟消息 message.setDelayTimeLevel(3); //messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h

五、批量消息 producer.send(messages);

消息大小限制在4M内， 应该有相同的Topic、waitStoreMsgOK、不能是延迟消息、事务消息

六、过滤消息（服务端broker过滤） Tag过滤 和 SQL过滤

broker 添加支持： enablePropertyFilter=true

tag

consumer.subscribe("TagFilterTest", "TagA || TagC");

sql

product

msg.putUserProperty("a", String.valueOf(i));

consumer

consumer.subscribe("SqlFilterTest", MessageSelector.bySql("(TAGS is not null and TAGS in ('TagA', 'TagB'))" + "and (a is not null and a between 0 and 3)"));

SQL92语法：

RocketMQ只定义了一些基本语法来支持这个特性

数值比较，比如：>，>=，<，<=，BETWEEN，=；
字符比较，比如：=，<>，IN；
IS NULL`` 或者 IS NOT NULL；
逻辑符号 AND，OR，NOT；

常量支持类型为：
数值，比如：123，3.1415；
字符，比如：'abc'，必须用单引号包裹起来；
NULL，特殊的常量
布尔值，TRUE 或 FALSE

七、事务消息 保证本地事务执行与消息发送两个操作的原子性，也就是这两个操作一起成功或者一起失败, 通过TransactionListener事务监听器控制

1. 事务消息不支持延迟消息和批量消息
2. 单个消息的检查次数限制为 15 次，但是用户可以通过 Broker 配置文件的 transactionCheckMax参数来修改此限制。如果已经检查某条消息超过 N 次的话（ N = transactionCheckMax ） 则 Broker 将丢弃此消息，并在默认情况下同时打印错误日志。用户可以通过重写 AbstractTransactionCheckListener 类来修改这个行为
3. 事务消息将在 Broker 配置文件中的参数 transactionMsgTimeout 这样的特定时间长度之后被检查。当发送事务消息时，用户还可以通过设置用户属性 CHECK_IMMUNITY_TIME_IN_SECONDS 来改变这个限制，该参数优先于 transactionMsgTimeout 参数
4. 事务性消息可能不止一次被检查或消费
5. 提交给用户的目标主题消息可能会失败，目前这依日志的记录而定。它的高可用性通过 RocketMQ 本身的高可用性机制来保证，如果希望确保事务消息不丢失、并且事务完整性得到保证，建议使用同步的双重写入机制
6. 务消息的生产者 ID 不能与其他类型消息的生产者 ID 共享。与其他类型的消息不同，事务消息允许反向查询、MQ服务器能通过它们的生产者 ID 查询到消费者

SendResult sendResult = producer.sendMessageInTransaction(msg, null);

TransactionListener transactionListener = new TransactionListenerImpl();
ExecutorService executorService = new ThreadPoolExecutor(2, 5, 100, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(2000), new ThreadFactory() {
    @Override
    public Thread newThread(Runnable r) {
        Thread thread = new Thread(r);
        thread.setName("client-transaction-msg-check-thread");
        return thread;
    }
});

producer.setExecutorService(executorService);
producer.setTransactionListener(transactionListener);

public interface TransactionListener {
    //执行本地事务
    LocalTransactionState executeLocalTransaction(Message var1, Object var2);
    //检查本地事务
    LocalTransactionState checkLocalTransaction(MessageExt var1);
}

• LocalTransactionState.COMMIT_MESSAGE；
• LocalTransactionState.ROLLBACK_MESSAGE;
• LocalTransactionState.UNKNOW;

事务消息机制的关键是在发送消息时，会将消息转为一个half半消息，并存入RocketMQ内部的一个 RMQ_SYS_TRANS_HALF_TOPIC 这个Topic，这样对消费者是不可见的。再经过一系列事务检查通过后，再将消息转存到目标Topic，这样对消费者就可见了

八、ACL权限控制

​ 权限控制（ACL）主要为RocketMQ提供Topic资源级别的用户访问控制

用户在使用RocketMQ权限控制时，可以在Client客户端通过 RPCHook注入AccessKey和SecretKey签名；

将对应的权限控制属性（包括Topic访问权限、IP白名单和AccessKey和SecretKey签名等）设置在$ROCKETMQ_HOME/conf/plain_acl.yml的配置文件中

Broker端对AccessKey所拥有的权限进行校验，校验不过，抛出异常

Broker端具体的配置信息可以参见 https://github.com/apache/rocketmq/blob/develop/docs/cn/acl/user_guide.md

1. 主要是在broker.conf中打开acl的标志：aclEnable=true

2. 权限配置 plain_acl.yml, 配置文件是热加载, 不用重启Broker服务

#全局白名单，不受ACL控制
#通常需要将主从架构中的所有节点加进来
globalWhiteRemoteAddresses:
- 10.10.103.*
- 192.168.0.*

accounts:
#第一个账户
- accessKey: RocketMQ
  secretKey: 12345678
  whiteRemoteAddress:
  admin: false 
  defaultTopicPerm: DENY #默认Topic访问策略是拒绝
  defaultGroupPerm: SUB #默认Group访问策略是只允许订阅
  topicPerms:
  - topicA=DENY #topicA拒绝
  - topicB=PUB|SUB #topicB允许发布和订阅消息
  - topicC=SUB #topicC只允许订阅
  groupPerms:
  # the group should convert to retry topic
  - groupA=DENY
  - groupB=PUB|SUB
  - groupC=SUB
#第二个账户，只要是来自192.168.1.*的IP，就可以访问所有资源
- accessKey: rocketmq2
  secretKey: 12345678
  whiteRemoteAddress: 192.168.1.*
  # if it is admin, it could access all resources
  admin: true

static RPCHook getAclRPCHook() {
    return new AclClientRPCHook(new SessionCredentials(ACL_ACCESS_KEY,ACL_SECRET_KEY));
}

//producer
DefaultMQProducer producer = new DefaultMQProducer("ProducerGroupName", getAclRPCHook());

//consumer
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_5", getAclRPCHook(), new AllocateMessageQueueAveragely());

DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("please_rename_unique_group_name_6", getAclRPCHook());

SpringBoot

https://github.com/apache/rocketmq-spring

pom.xml

 <dependency>
     <groupId>org.apache.rocketmqgroupId>
     <artifactId>rocketmq-spring-boot-starterartifactId>
     <version>2.2.1version>
     <exclusions>
         <exclusion>
             <groupId>org.springframework.bootgroupId>
             <artifactId>spring-boot-starterartifactId>
         exclusion>
         <exclusion>
             <groupId>org.springframeworkgroupId>
             <artifactId>spring-coreartifactId>
         exclusion>
         <exclusion>
             <groupId>org.springframeworkgroupId>
             <artifactId>spring-webmvcartifactId>
         exclusion>
     exclusions>
 dependency>

application.properties

rocketmq.name-server=rocketmq-2.localhost.com:9876
rocketmq.producer.group=springBootGroup

消息生产者 RocketMQTemplate

@Component
public class SpringProducer {

    @Resource
    private RocketMQTemplate rocketMQTemplate;

    public void sendMessage(String topic,String msg){
        this.rocketMQTemplate.convertAndSend(topic,msg);
    }

    public void sendMessageInTransaction(String topic,String msg) throws InterruptedException {
        String[] tags = new String[] {"TagA", "TagB", "TagC", "TagD", "TagE"};
        for (int i = 0; i < 10; i++) {
            //尝试在Header中加入一些自定义的属性。
            Message<String> message = MessageBuilder.withPayload(msg)
                    .setHeader(RocketMQHeaders.TRANSACTION_ID,"TransID_"+i)
                    //发到事务监听器里后，这个自己设定的TAGS属性会丢失。但是上面那个属性不会丢失。
                    .setHeader(RocketMQHeaders.TAGS,tags[i % tags.length])
                    //MyProp在事务监听器里也能拿到，为什么就单单这个RocketMQHeaders.TAGS拿不到？这只能去调源码了。
                    .setHeader("MyProp","MyProp_"+i)
                    .build();
            String destination =topic+":"+tags[i % tags.length];
            //这里发送事务消息时，还是会转换成RocketMQ的Message对象，再调用RocketMQ的API完成事务消息机制。
            SendResult sendResult = rocketMQTemplate.sendMessageInTransaction(destination, message,destination);
            System.out.printf("%s%n", sendResult);

            Thread.sleep(10);
        }
    }
}

事务监听器 SpringBoot的自带的事务ID不一定会一直 需要自己实现通过 setHeader("MyProp","MyProp_"+i) 设置， msg.getHeaders().get("MyProp") 获取

@RocketMQTransactionListener(rocketMQTemplateBeanName = "rocketMQTemplate")
public class MyTransactionImpl implements RocketMQLocalTransactionListener {

    private ConcurrentHashMap<Object, Message> localTrans = new ConcurrentHashMap<>();
    @Override
    public RocketMQLocalTransactionState executeLocalTransaction(Message msg, Object arg) {
        Object transId = msg.getHeaders().get(RocketMQHeaders.PREFIX+RocketMQHeaders.TRANSACTION_ID);
        String destination = arg.toString();
        localTrans.put(transId,msg);
        //这个msg的实现类是GenericMessage，里面实现了toString方法
        //在Header中自定义的RocketMQHeaders.TAGS属性，到这里就没了。但是RocketMQHeaders.TRANSACTION_ID这个属性就还在。
        //而message的Header里面会默认保存RocketMQHeaders里的属性，但是都会加上一个RocketMQHeaders.PREFIX前缀
        System.out.println("executeLocalTransaction msg = "+msg);
        //转成RocketMQ的Message对象
        org.apache.rocketmq.common.message.Message message = RocketMQUtil.convertToRocketMessage(new StringMessageConverter(),"UTF-8",destination, msg);
        String tags = message.getTags();
        if(StringUtils.contains(tags,"TagA")){
            return RocketMQLocalTransactionState.COMMIT;
        }else if(StringUtils.contains(tags,"TagB")){
            return RocketMQLocalTransactionState.ROLLBACK;
        }else{
            return RocketMQLocalTransactionState.UNKNOWN;
        }
    }
    //延迟检查的时间间隔要有点奇怪。
    @Override
    public RocketMQLocalTransactionState checkLocalTransaction(Message msg) {
        String transId = msg.getHeaders().get(RocketMQHeaders.PREFIX+RocketMQHeaders.TRANSACTION_ID).toString();
        Message originalMessage = localTrans.get(transId);
        //这里能够获取到自定义的transaction_id属性
        System.out.println("checkLocalTransaction msg = "+originalMessage);
        //获取标签时，自定义的RocketMQHeaders.TAGS拿不到，但是框架会封装成一个带RocketMQHeaders.PREFIX的属性
//        String tags = msg.getHeaders().get(RocketMQHeaders.TAGS).toString();
        String tags = msg.getHeaders().get(RocketMQHeaders.PREFIX+RocketMQHeaders.TAGS).toString();
        if(StringUtils.contains(tags,"TagC")){
            return RocketMQLocalTransactionState.COMMIT;
        }else if(StringUtils.contains(tags,"TagD")){
            return RocketMQLocalTransactionState.ROLLBACK;
        }else{
            return RocketMQLocalTransactionState.UNKNOWN;
        }
    }
}

由于事件监听器需要通过rocketMQTemplateBeanName名字来识别不同Bean

@ExtRocketMQTemplateConfiguration()
public class ExtRocketMQTemplate extends RocketMQTemplate {}

消息消费者 @RocketMQMessageListener

@Component
@RocketMQMessageListener(consumerGroup = "MyConsumerGroup", topic = "TestTopic",consumeMode= ConsumeMode.CONCURRENTLY)
public class SpringConsumer implements RocketMQListener<String> {
    @Override
    public void onMessage(String message) {
        System.out.println("Received message : "+ message);
    }
}

RocketMQMessageListener

public @interface RocketMQMessageListener {
    String NAME_SERVER_PLACEHOLDER = "${rocketmq.name-server:}";
    String ACCESS_KEY_PLACEHOLDER = "${rocketmq.consumer.access-key:}";
    String SECRET_KEY_PLACEHOLDER = "${rocketmq.consumer.secret-key:}";
    String TRACE_TOPIC_PLACEHOLDER = "${rocketmq.consumer.customized-trace-topic:}";
    String ACCESS_CHANNEL_PLACEHOLDER = "${rocketmq.access-channel:}";

    String consumerGroup();

    String topic();

	//TAG, SQL92
    SelectorType selectorType() default SelectorType.TAG;

    String selectorExpression() default "*";

	//CONCURRENTLY,ORDERLY;
    ConsumeMode consumeMode() default ConsumeMode.CONCURRENTLY;

	//BROADCASTING,CLUSTERING;
    MessageModel messageModel() default MessageModel.CLUSTERING;

    int consumeThreadMax() default 64;

    int maxReconsumeTimes() default -1;

    long consumeTimeout() default 15L;

    int replyTimeout() default 3000;

    String accessKey() default "${rocketmq.consumer.access-key:}";

    String secretKey() default "${rocketmq.consumer.secret-key:}";

    boolean enableMsgTrace() default false;

    String customizedTraceTopic() default "${rocketmq.consumer.customized-trace-topic:}";

    String nameServer() default "${rocketmq.name-server:}";

    String accessChannel() default "${rocketmq.access-channel:}";
}

Test

@RunWith(SpringRunner.class)
@SpringBootTest
public class SpringRocketTest {

    @Resource
    private RocketMQTemplate rocketMQTemplate;

    @Test
    public void sendMessageTest(){
        String springTopic="TestTopic";
        //发送字符消息
        SendResult sendResult = rocketMQTemplate.syncSend(springTopic, "Hello, World!");
        System.out.printf("syncSend1 to topic %s sendResult=%s %n", springTopic, sendResult);

        sendResult = rocketMQTemplate.syncSend(springTopic, new User().setUserAge((byte) 18).setUserName("Kitty"));
        System.out.printf("syncSend1 to topic %s sendResult=%s %n", springTopic, sendResult);

        sendResult = rocketMQTemplate.syncSend(springTopic, MessageBuilder.withPayload(
                new User().setUserAge((byte) 21).setUserName("Lester")).setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON_VALUE).build());
        System.out.printf("syncSend1 to topic %s sendResult=%s %n", springTopic, sendResult);

        //发送对象消息
        rocketMQTemplate.asyncSend(springTopic, new OrderPaidEvent("T_001", new BigDecimal("88.00")), new SendCallback() {
            @Override
            public void onSuccess(SendResult var1) {
                System.out.printf("async onSucess SendResult=%s %n", var1);
            }

            @Override
            public void onException(Throwable var1) {
                System.out.printf("async onException Throwable=%s %n", var1);
            }
        });

        //发送指定TAG的消息
        rocketMQTemplate.convertAndSend(springTopic + ":tag0", "I'm from tag0");  // tag0 will not be consumer-selected
        System.out.printf("syncSend topic %s tag %s %n", springTopic, "tag0");
        rocketMQTemplate.convertAndSend(springTopic + ":tag1", "I'm from tag1");
        System.out.printf("syncSend topic %s tag %s %n", springTopic, "tag1");

        //同步发送消息并且返回一个String类型的结果。
        String replyString = rocketMQTemplate.sendAndReceive(springTopic, "request string", String.class);
        System.out.printf("send %s and receive %s %n", "request string", replyString);

        //同步发送消息并且返回一个Byte数组类型的结果。
        byte[] replyBytes = rocketMQTemplate.sendAndReceive(springTopic, MessageBuilder.withPayload("request byte[]").build(), byte[].class, 3000);
        System.out.printf("send %s and receive %s %n", "request byte[]", new String(replyBytes));

        //同步发送一个带hash参数的请求(排序消息)，并返回一个User类型的结果
        User requestUser = new User().setUserAge((byte) 9).setUserName("requestUserName");
        User requestUser2 = new User().setUserAge((byte) 9).setUserName("requestUserName");
        User requestUser3 = new User().setUserAge((byte) 9).setUserName("requestUserName");
        User requestUser4 = new User().setUserAge((byte) 9).setUserName("requestUserName");
        User replyUser = rocketMQTemplate.sendAndReceive(springTopic, requestUser, User.class, "order-id");
        User replyUser2 = rocketMQTemplate.sendAndReceive(springTopic, requestUser2, User.class, "order-id");
        User replyUser3 = rocketMQTemplate.sendAndReceive(springTopic, requestUser3, User.class, "order-id");
        User replyUser4 = rocketMQTemplate.sendAndReceive(springTopic, requestUser4, User.class, "order-id");
        System.out.printf("send %s and receive %s %n", requestUser, replyUser);
        //同步发送一个带延迟级别的消息(延迟消息)，并返回一个泛型结果
        ProductWithPayload<String> replyGenericObject = rocketMQTemplate.sendAndReceive(springTopic, "request generic",
                new TypeReference<ProductWithPayload<String>>() {
                }.getType(), 30000, 2);
        System.out.printf("send %s and receive %s %n", "request generic", replyGenericObject);

        //异步发送消息，返回String类型结果。
        rocketMQTemplate.sendAndReceive(springTopic, "request string", new RocketMQLocalRequestCallback<String>() {
            @Override public void onSuccess(String message) {
                System.out.printf("send %s and receive %s %n", "request string", message);
            }

            @Override public void onException(Throwable e) {
                e.printStackTrace();
            }
        });
        //异步发送消息，并返回一个User类型的结果。
        rocketMQTemplate.sendAndReceive(springTopic, new User().setUserAge((byte) 9).setUserName("requestUserName"), new RocketMQLocalRequestCallback<User>() {
            @Override public void onSuccess(User message) {
                System.out.printf("send user object and receive %s %n", message.toString());
            }

            @Override public void onException(Throwable e) {
                e.printStackTrace();
            }
        }, 5000);
        //发送批量消息
        List<Message> msgs = new ArrayList<Message>();
        for (int i = 0; i < 10; i++) {
            msgs.add(MessageBuilder.withPayload("Hello RocketMQ Batch Msg#" + i).
                    setHeader(RocketMQHeaders.KEYS, "KEY_" + i).build());
        }

        SendResult sr = rocketMQTemplate.syncSend(springTopic, msgs, 60000);

        System.out.printf("--- Batch messages send result :" + sr);
    }
}