RocketMQ学习

一、角色

1.NameServer

• 服务的注册与发现、路由管理。类似于zookeeper

2.BrokerCluster

• broker面向producer和consumer接受和发送消息
• 向nameserver提供自己的消息
• 负责消息的存储

3.producer

• 消息生产者

4.consumer

• 消息发送者

参考文章：https://blog.csdn.net/weixin_42923363/article/details/123111414

二、基本概念

• 和kafka的基本概念类似，分为consumerGroup，topic，messageQueue（对应kafka为partition），offset。
• rmq需要一个producerGroup，与consumerGroup为同一个
• tag可以为topic中的queue打上标签（kafka里面好像没有）

三、docker安装rmq

• 可以借鉴大佬的博客安装

https://blog.csdn.net/ming19951224/article/details/109063041

四、先验证简单发送消息

1.导入依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>rmq</artifactId>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.5.6</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
    </properties>

    <dependencies>
        <!--springboot-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <!--lombok-->
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.4</version>
            <scope>provided</scope>
        </dependency>
        <!--rocketmq-->
        <dependency>
            <groupId>org.apache.rocketmq</groupId>
            <artifactId>rocketmq-spring-boot-starter</artifactId>
            <version>2.1.1</version>
        </dependency>

        <!--        引入log4j2依赖 -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-log4j2</artifactId>
        </dependency>
        <dependency>
            <groupId>com.lmax</groupId>
            <artifactId>disruptor</artifactId>
            <version>3.4.2</version>
        </dependency>
        <!--        引入log4j2依赖 -->
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

2.springboot配置文件：application.xml

server:
  port: 8088

rocketmq:
#这是你的nameserver的ip和端口
  name-server: 127.0.0.1:9876
  producer:
    group: dev-group
    topic: dev

  consumer:
    topic: dev
    group: dev-group

3.生产者代码

package com.rmq.producer;

import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

/**
 * @Author: towards
 * @CreateTime: 2022-07-02  11:11
 * @Description: TODO
 * @Version: 1.0
 */
@Slf4j
@Service
public class SimpleProducer {

    @Value("${rocketmq.producer.topic}")
    private String topic;

    @Autowired
    private RocketMQTemplate rocketMQTemplate;

    /**
     * @description: 简单的生产消息
     * @author: towards
     * @date: 2022/7/2 11:29
     **/
    public void syncProducer(){
        for (int i = 0; i < 10; i++) {
            rocketMQTemplate.convertAndSend(topic,"this is message"+i);
        }
    }
}

4.消费者代码

package com.rmq.consumer;

import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
import org.apache.rocketmq.spring.core.RocketMQListener;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;


/**
 * @Author: towards
 * @CreateTime: 2022-07-02  11:10
 * @Description: 消费者
 * @Version: 1.0
 */
@Slf4j
@Service
@RocketMQMessageListener(topic = "dev",consumerGroup = "dev-group")
public class ListenMsg implements RocketMQListener<String> {

    /**
     * @description: 监听topic消息
     * @author: towards
     * @date: 2022/7/2 12:15
     * @param: message
     **/
    @Override
    public void onMessage(String message) {
        log.info("topic : dev , group : dev-group . message : {}",message);
    }
}

5.控制台展示消息

五、前置准备

1.准备一个controller

• 目的是调用以下生产消息的方法，以便于及时消费消息
• 更重要的是，有些发送消息的回调函数，单元测试不能获取到。原因就是像异步的这种情况，主线程已经走完全部流程了，异步的线程还没获取到回调函数，所以无法打印回调函数里面的消息

package com.rmq.controller;

import com.rmq.producer.AsyncProducer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

/**
 * @Author: towards
 * @CreateTime: 2022-07-02  15:21
 * @Description: TODO
 * @Version: 1.0
 */
@RestController
public class RmqController {

    @Autowired
    AsyncProducer asyncProducer;

    @GetMapping("async")
    public String asyncProduc(){
        asyncProducer.producAsyncMsg();
        return "async";
    }

}

2.准备一个SendCallbackListener工具

• 主要实现异步消息的回调（注意，单元测试并不会走此回调函数）

package com.rmq.rmqutil;
 
import com.alibaba.fastjson.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.client.producer.SendCallback;
import org.apache.rocketmq.client.producer.SendResult;
 
/**
 * rocketmq异步回调监听
 */
@Slf4j
public class SendCallbackListener implements SendCallback {
 
    private int id;
 
    public SendCallbackListener(int id) {
        this.id = id;
    }
 
    @Override
    public void onSuccess(SendResult sendResult) {
        log.info("CallBackListener on success : " + JSONObject.toJSONString(sendResult));
    }
 
    @Override
    public void onException(Throwable throwable) {
        log.error("CallBackListener on exception : ", throwable);
    }
}

六、发送同步消息

1.配置文件

server:
  port: 8088

rocketmq:
  name-server: 127.0.0.1:9876
  producer:
    group: dev-group
    topic: dev
    sync-tag: sync-tag
    async-tag: async-tag
    oneway-tag: oneway-tag

  consumer:
    topic: dev
    group: dev-group

2.同步消息生产者

• 给topic：dev加上同步消息的标签就可以

package com.rmq.producer;

import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.client.producer.SendStatus;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Service;

import java.util.Random;

/**
 * @Author: towards
 * @CreateTime: 2022-07-02  14:08
 * @Description: 生产同步消息
 * @Version: 1.0
 */
@Service
@Slf4j
public class SyncProducer {

    @Autowired
    private RocketMQTemplate rocketMQTemplate;

    @Value("${rocketmq.producer.topic}:${rocketmq.producer.sync-tag}")
    private String syncTag;

    public void producSyncMsg(){
        // 构建消息
        Random random = new Random();
        int i = random.nextInt();
        String msg = "sync message "+ i;
        // withPayload 载荷，表示发送的消息
        // setHeader 头部，表示该条消息的头部
        Message<String> message = MessageBuilder
                .withPayload(msg)
                .setHeader(RocketMQHeaders.KEYS, i)
                .build();
        log.info("message : {}",message);
        // sendResult 表示发送消息的响应状态
        SendResult sendResult = rocketMQTemplate.syncSend(syncTag, message);
        log.info("sendResult : {}",sendResult);
        if (sendResult.getSendStatus() == SendStatus.SEND_OK){
            log.info("send sync message success!");
        }
    }

}

3.控制台展示消息

• 会发现之前的topic下面的消息，有一条被加上了tag和key
  

七、发送异步消息

1.异步消息生产者

package com.rmq.producer;

import com.rmq.rmqutil.SendCallbackListener;
import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.client.producer.SendCallback;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Service;

/**
 * @Author: towards
 * @CreateTime: 2022-07-02  14:32
 * @Description: 发送异步消息
 * @Version: 1.0
 */
@Service
@Slf4j
public class AsyncProducer {

    @Autowired
    RocketMQTemplate rocketMQTemplate;

    @Value("${rocketmq.producer.topic}:${rocketmq.producer.async-tag}")
    private String asyncTag;

    public void producAsyncMsg(){
        Integer id = 1;
        String msg = "this is async msg!";
        Message<String> message = MessageBuilder.withPayload(msg)
                .setHeader(RocketMQHeaders.KEYS, id)
                .build();
        rocketMQTemplate.asyncSend(asyncTag, message, new SendCallbackListener(id));
        log.info("send async message finish!");
    }

}



}

2.观察日志可以发现异步线程打印回调函数在主线程之后

八、发送单向消息

• 发送单向消息，不需要响应，一般记录日志

1.单向消息生产者

package com.rmq.producer;

import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Service;

/**
 * @Author: towards
 * @CreateTime: 2022-07-02  15:06
 * @Description: 发送单向消息，不需要响应，一般记录日志
 * @Version: 1.0
 */
@Slf4j
@Service
public class SingleProducer {

    @Autowired
    private RocketMQTemplate rocketMQTemplate;

    @Value("${rocketmq.producer.topic}:${rocketmq.producer.oneway-tag}")
    private String oneWayTag;

    public void produceSingleMsg(){
        Integer id = 1;
        String msg = "this is oneway message";
        Message<String> message = MessageBuilder.withPayload(msg)
                .setHeader(RocketMQHeaders.KEYS, 1)
                .build();
        log.info("send oneway message : {} , to topic : {}",message,oneWayTag);
        rocketMQTemplate.sendOneWay(oneWayTag,message);
    }

}

九、发送顺序同步消息

1.为什么要顺序发送消息

• rmq有序分为分区有序和全局有序。
• 在默认的情况下，发送者将消息轮询发送到rmq的所有分区队列queue，消费者从多个queue上拉取消息，但此时并不能保证发送和消费的顺序。
• 如果指定消息发送到指定的queue，则消费者消费该queue时，能够保证分区有序。
• 比如在创建一个订单时，下单、支付、发货、完成，此时需要保证消息消费的顺序，则发送消息指定为同一个queue

2.顺序同步消息生产者

package com.rmq.producer;

import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.client.producer.MessageQueueSelector;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Service;
import sun.rmi.runtime.Log;

import java.util.ArrayList;
import java.util.List;
import java.util.UUID;

/**
 * @Author: towards
 * @CreateTime: 2022-07-02  15:56
 * @Description: 包含顺序的同步消息
 * @Version: 1.0
 */
@Service
@Slf4j
public class OrderSyncProducer {

    @Autowired
    RocketMQTemplate rocketMQTemplate;

    @Value("${rocketmq.producer.topic}:${rocketmq.producer.sync-tag}")
    private String syncTag;

    public void produceOrderSync(){
        // 创建三个消息
        String msg = "order syncOrder message ";
        List<String> msgList = new ArrayList<>();
        for (int i = 0; i < 3; i++) {
            msgList.add(msg+i);
        }
        log.info("msgList is : {}",msgList);
        // 依次发消息
        for (String message : msgList){
            // 构建消息
            String msgStr = String.format("order id : %s , pay : %s",msgList.size()+1,message);
            Message<String> rmqMsg = MessageBuilder.withPayload(msgStr)
                    .setHeader(RocketMQHeaders.KEYS, msgList.size() + 1)
                    .build();
            log.info("rmqMsg is : {}",rmqMsg);
            // 创建消息队列选择器，设置顺序下发
            rocketMQTemplate.setMessageQueueSelector(new MessageQueueSelector() {
                /**
                 * 设置放入同一个队列的规则
                 * @param list 消息列表
                 * @param message 当前消息
                 * @param o 比较的关键信息
                 * @return 消息队列
                 */
                @Override
                public MessageQueue select(List<MessageQueue> list, org.apache.rocketmq.common.message.Message message, Object o) {
                    // 根据当前消息的id，使用固定算法获取需要下发的队列
                    // （使用当前id和消息队列个数进行取模获取需要下发的队列，id和队列数量一样时，选择的队列肯定一样）
                    int queueNum = Integer.valueOf(msgList.hashCode()) % list.size();
                    log.info(String.format("queueNum : %s, rmqMsg : %s", queueNum, new String(message.getBody())));
                    return list.get(queueNum);
                }
            });
            SendResult sendResult = rocketMQTemplate.syncSendOrderly(syncTag, rmqMsg, message);
            log.info("order msg sendResult is : {}",sendResult);
        }
    }
}

十、延迟消息

1.延迟消息生产者

package com.rmq.producer;

import com.alibaba.fastjson.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Service;

import java.util.ArrayList;
import java.util.List;

/**
 * @Author: towards
 * @CreateTime: 2022-07-02  20:47
 * @Description: 延迟消息
 * @Version: 1.0
 */
@Service
@Slf4j
public class DelayProducer {

    @Autowired
    private RocketMQTemplate rocketMQTemplate;

    @Value("${rocketmq.producer.topic}:${rocketmq.producer.sync-tag}")
    private String syncTag;

    public void produceDelay(){
        // 创建消息
        String msg = "this is delay message ";
        Message<String> message = MessageBuilder.withPayload(msg)
                .setHeader(RocketMQHeaders.KEYS, 1)
                .build();
        // 设置超时和延时推送
        // 超时时针对请求broker然后结果返回给product的耗时
        // 现在RocketMq并不支持任意时间的延时，需要设置几个固定的延时等级，从1s到2h分别对应着等级1到18
        // private String messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h";
        SendResult sendResult = rocketMQTemplate.syncSend(syncTag, message, 1 * 1000l, 4);
        log.info("pushDelayMessage : " + message + ", sendResult : " + JSONObject.toJSONString(sendResult));
    }
}

十一、批量发送消息

• 之前发送多条消息并非批量，只是多次调用RocketMQTemplate循环发送
• 真正的批量发送应该是组装成一个消息集合，调用RocketMQTemplate一次发送多条消息

1.批量消息生产者

package com.rmq.producer;

import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Service;

import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;

/**
 * @Author: towards
 * @CreateTime: 2022-07-02  21:10
 * @Description: TODO
 * @Version: 1.0
 */
@Slf4j
@Service
public class BatchProducer {

    @Autowired
    private RocketMQTemplate rocketMQTemplate;

    @Value("${rocketmq.producer.topic}:${rocketmq.producer.sync-tag}")
    private String syncTag;

    public void produceBatchMsg(){
        List<Message> rmqMsgs = new ArrayList<>();
        for (int i = 0; i < 10; i++) {
            String msgStr = "this is batch message , message num is : "+i;
            Message<String> rmqMsg = MessageBuilder.withPayload(msgStr)
                    .setHeader(RocketMQHeaders.KEYS, i)
                    .build();
            rmqMsgs.add(rmqMsg);
        }
        log.info("if haven't filter msg , rmqMsgs is : {}",rmqMsgs.toString());
        // 批量下发消息到broker,不支持消息顺序操作，不支持异步消息，并且对消息体有大小限制（不超过4M）
        // 可以对消息进行过滤，这里只是对消息进行长度判断，并不是对判断字节大小
        List<Message> messages = rmqMsgs.stream().filter(x -> x.getPayload().toString().length() < 1024 * 1024 * 1024).collect(Collectors.toList());
        SendResult sendResult = rocketMQTemplate.syncSend(syncTag, messages);
        log.info("send batchSyncMsg sendResult is : {}",sendResult);
    }
}

十二、发送事务消息

• 发送事务消息的前提，需要一个事务消费者

1.事务消费者

package com.rmq.consumer;
 
import com.alibaba.fastjson.JSONObject;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.client.producer.SendCallback;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.spring.annotation.RocketMQTransactionListener;
import org.apache.rocketmq.spring.core.RocketMQLocalTransactionListener;
import org.apache.rocketmq.spring.core.RocketMQLocalTransactionState;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHeaders;
 
import java.util.HashMap;
import java.util.Map;
 
/**
 * rocketmq异步回调监听
 */
@Slf4j
@RocketMQTransactionListener
@RequiredArgsConstructor(onConstructor = @__(@Autowired))
public class TransactionListener implements RocketMQLocalTransactionListener {
    private static final Map<String, RocketMQLocalTransactionState> TRANSACTION_STATE_MAP = new HashMap<>();
 
    /**
     * 处理本地事务
     * @param message
     * @param o
     * @return
     */
    @Override
    public RocketMQLocalTransactionState executeLocalTransaction(Message message, Object o) {
        log.info("执行本地事务");
        MessageHeaders headers = message.getHeaders();
        //获取事务ID
        String transactionId = (String) headers.get(RocketMQHeaders.TRANSACTION_ID);
        TRANSACTION_STATE_MAP.put(transactionId, RocketMQLocalTransactionState.UNKNOWN);
        log.info("transactionId is {}",transactionId);
        try {
            Thread.sleep(10 * 1000l);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        RocketMQLocalTransactionState state = RocketMQLocalTransactionState.ROLLBACK;
        if (Integer.parseInt(transactionId) % 2 == 0) {
            //执行成功，可以提交事务
            state = RocketMQLocalTransactionState.COMMIT;
        }
        log.info("transactionId is {}, state {}",transactionId, state.toString());
        TRANSACTION_STATE_MAP.remove(transactionId);
        return state;
    }
 
    /**
     * 校验事务状态
     * @param message
     * @return
     */
    @Override
    public RocketMQLocalTransactionState checkLocalTransaction(Message message) {
        MessageHeaders headers = message.getHeaders();
        //获取事务ID
        String transactionId = (String) headers.get(RocketMQHeaders.TRANSACTION_ID);
        log.info("检查本地事务,事务ID:{}",transactionId);
        RocketMQLocalTransactionState state = TRANSACTION_STATE_MAP.get(transactionId);
        if(null != state){
            return state;
        }
        return RocketMQLocalTransactionState.ROLLBACK;
 
    }
}

2.事务消息生产者

package com.rmq.producer;

import lombok.extern.slf4j.Slf4j;
import org.apache.rocketmq.client.producer.TransactionSendResult;
import org.apache.rocketmq.spring.core.RocketMQTemplate;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.stereotype.Service;

/**
 * @Author: towards
 * @CreateTime: 2022-07-02  21:34
 * @Description: 事务消息
 * @Version: 1.0
 */
@Slf4j
@Service
public class TransactionProducer {

    @Autowired
    private RocketMQTemplate rocketMQTemplate;

    @Value("${rocketmq.producer.topic}:${rocketmq.producer.sync-tag}")
    private String syncTag;

    public void producerTransactionMsg(){
        String msg = "this is transaction message!";
        Message<String> message = MessageBuilder.withPayload(msg)
                .setHeader(RocketMQHeaders.TRANSACTION_ID, 2)
                .setHeader(RocketMQHeaders.KEYS,2)
                .build();
        TransactionSendResult transactionSendResult = rocketMQTemplate.sendMessageInTransaction(syncTag, message, null);
        log.info("send transaction message , sendResult is : {}",transactionSendResult);
    }
}

本文引用大佬的博客，写的非常清晰明了，要是有不懂的地方可以参考大佬原文：
大佬原文链接