如何使用RabbitMQ中的死信交换器(dead letter exchange)

RabbitMQ中的死信交换器(dead letter exchange)可以接收下面三种场景中的消息:

• 消费者对消息使用了basicReject或者basicNack回复,并且requeue参数设置为false,即不再将该消息重新在消费者间进行投递.
• 消息在队列中超时. RabbitMQ可以在单个消息或者队列中设置TTL属性.
• 队列中的消息已经超过其设置的最大消息个数.

死信交换器可以在程序中设置,也可以使用rabbitmqctl工具进行设置.关于死信交换器的介绍请参考RabbitMQ官网https://www.rabbitmq.com/dlx.html.

死信交换器可以统计或者监控错误消息, 下面使用RabbitMQ的Java客户端库给出一个完整的例子, 结合第一种消息来源说明死信交换器的使用.

示例代码

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;

import com.rabbitmq.client.AMQP.BasicProperties;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.DeliverCallback;
import com.rabbitmq.client.Envelope;

public class DeadLetterExchange {
    private static final String NORMAL_EXCHANGE = "normal_exchange";
    private static final String DEAD_LETTER_EXCHANGE = "dead_letter_exchange";    
    private static final String DEFAULT_WORKER_QUEUE = "queue_to_normal_exchange";
    private static final String DLX_WORKER_QUEUE = "queue_to_deadletter_exchange";
    private static final String NORMAL_ROUTE_KEY = "normal_route_key";
    private static final String DLX_ROUTE_KEY = "dlx_route_key";
        
    private Connection connection;
    // For consumer and producer schedule.
    private final ExecutorService executorService;
    
    private static class SimpleThreadFactory implements ThreadFactory {
        private final AtomicInteger idx = new AtomicInteger(0);
                
        public Thread newThread(Runnable r) {
            return new Thread(r, "Thread#" + idx.getAndIncrement());            
        }        
    }
    
    private static class WorkerConsumer implements Runnable {        
        private final Connection connection;
        private Channel channel;
        private String queue;
        
        private final DeliverCallback deliverCallback = (consumerTag, delivery) -> {            
            final String message = new String(delivery.getBody(), "UTF-8");
            final Envelope envelope = delivery.getEnvelope();
            final long deliveryTag = envelope.getDeliveryTag();
            final boolean redelivery = envelope.isRedeliver();
            final BasicProperties props = delivery.getProperties();
            final String msgId = props.getMessageId();
            
            // Reject some message and not requeue it.            
            final boolean reject = (deliveryTag % 3) == 2;
            System.out.println("Received normal exchange msg:" + message 
                            + " msg id:" + msgId 
                            + " reject:" + reject);
            if (reject ) {
                channel.basicReject(deliveryTag, false);
            } else {
                channel.basicAck(deliveryTag, false);
            }                       
        };
        
        public WorkerConsumer(Connection connection, String queue) {
            this.connection = connection; 
            this.queue = queue;
        }
        
        @Override
        public void run() {
            try {
                channel = connection.createChannel();                
                channel.exchangeDeclare(NORMAL_EXCHANGE, "direct");                
                final Map args = new HashMap<>();
                args.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE);
                if (queue == null) {
                    queue = DEFAULT_WORKER_QUEUE;
                }
                channel.queueDeclare(queue, false, false, true, args);
                channel.queueBind(queue, NORMAL_EXCHANGE, NORMAL_ROUTE_KEY);                
                final String tag = channel.basicConsume(queue, false, deliverCallback, consumerTag -> {System.out.println("Cancel worker consumer:" +consumerTag);});
                System.out.println("worker consumer tag:" + tag);
            } catch (IOException e) {
                e.printStackTrace();                
            }                   
        }        
    }
    
    // This special consumer handle messages delivered to dead letter exchange.
    private static class DLXConsumer implements Runnable {        
        private final Connection connection;
        
        private final DeliverCallback deliverCallback = (consumerTag, delivery) -> {            
            final String message = new String(delivery.getBody(), "UTF-8");
            final BasicProperties props = delivery.getProperties();
            final String msgId = props.getMessageId();
            System.out.println("Received dead letter exchange msg:" + message 
                            + " msg id:" + msgId);
        };
        
        public DLXConsumer(Connection connection) {
            this.connection = connection;            
        }
        
        @Override
        public void run() {
            try {
                final Channel channel = connection.createChannel();                
                channel.exchangeDeclare(DEAD_LETTER_EXCHANGE, "direct");                                       
                channel.queueDeclare(DLX_WORKER_QUEUE, false, false, true, null);
                channel.queueBind(DLX_WORKER_QUEUE, DEAD_LETTER_EXCHANGE, NORMAL_ROUTE_KEY);                
                final String tag = channel.basicConsume(DLX_WORKER_QUEUE, true, deliverCallback, consumerTag -> {System.out.println("Cancel DLX consumer:" +consumerTag);});
                System.out.println("dlx consumer tag:" + tag);
            } catch (IOException e) {
                e.printStackTrace();
            }                     
        } 
    }
    
    private static class Producer implements Runnable {
        private static final int MSG_COUNTS = 20;
        private final Connection connection;
        
        public Producer(Connection connection) {
            this.connection = connection;            
        }
        
        @Override
        public void run() {
            try {
                final Channel channel = connection.createChannel();                                                       
                // Send some messages, some of them can not route to any queue.
                for (int i = 0; i < MSG_COUNTS; i++) {
                    final BasicProperties.Builder propsBuilder = new BasicProperties.Builder();
                    final BasicProperties props = propsBuilder.appId(Thread.currentThread().toString()).messageId(String.valueOf(i)).build();                                        
                    final boolean useInvalidRouteKey = (i % 4 == 3);
                    if (!useInvalidRouteKey) {
                        channel.basicPublish(NORMAL_EXCHANGE, NORMAL_ROUTE_KEY, props, "[normal message.]".getBytes());
                    } else {
                        channel.basicPublish(NORMAL_EXCHANGE, "not_existed_route_key", props, "[lost message.]".getBytes());
                    }                    
                }                
            } catch(IOException e) {
                e.printStackTrace();
            }
        }        
    }
    
    public DeadLetterExchange() {
        executorService = Executors.newCachedThreadPool(new SimpleThreadFactory());
        
        final ConnectionFactory connectionFactory = new ConnectionFactory();
        connectionFactory.setHost("localhost");
        try {
            connection = connectionFactory.newConnection();
        } catch (IOException | TimeoutException e) {
            e.printStackTrace();
        }        
    }
    
    public void execWorkderConsumer(String queue) {
        executorService.submit(new WorkerConsumer(connection, queue));               
    }

    public void execDLXConsumer() {
        executorService.submit(new DLXConsumer(connection));
    }    
    
    public void execProducer(int n) {
        for (int i = 0; i < n; i++) {
            executorService.submit(new Producer(connection));
        }        
    } 
    
    public void clearConfig(String[] queues) {
        try {
            final Channel channel = connection.createChannel();
            channel.exchangeDelete(DEAD_LETTER_EXCHANGE);
            channel.exchangeDelete(NORMAL_EXCHANGE);
            channel.queueDelete(DEFAULT_WORKER_QUEUE);
            channel.queueDelete(DLX_WORKER_QUEUE);
            for (String queue : queues) {
                channel.queueDelete(queue);
            }
            System.out.println("Config cleared.");
            System.exit(0);
        } catch(IOException e) {
            e.printStackTrace();            
        }        
    }
    
    public static void main(String args[]) {
        if (args.length > 0) {
            final String which = args[0];                    
            if ("consumer".equals(which)) {
                final String workQueue = (args.length > 1 ? args[1] : null);
                new DeadLetterExchange().execWorkderConsumer(workQueue);            
            } else if ("producer".equals(which)) {
                final int n = (args.length > 1 ? Integer.parseInt(args[1]) : 1);
                new DeadLetterExchange().execProducer(n);            
            } else if ("dlxconsumer".equals(which)) {
                new DeadLetterExchange().execDLXConsumer();
            } else if ("clearconfig".equals(which)) {
                String[] queues = new String[0];
                if (args.length > 1) {
                    queues = new String[args.length - 1];
                    System.arraycopy(args, 1, queues, 0, args.length - 1);
                }
                new DeadLetterExchange().clearConfig(queues);                
            }
        }                     
    }
}

其中,producer投递消息的交换器定义为NORMAL_EXCHANGE, 其consumer实现定义为WorkerConsumer, 死信交换器定义为DEAD_LETTER_EXCHANGE, 其consumer实现定义为DLXConsumer.

示例代码中,可以为NORMAL_EXCHANGE绑定实现一个消息队列, 可以运行多个consumer在该队列中以Round-robin的方式处理消息, 也可以为NORMAL_EXCHANGE绑定实现多个队列.通常死信交换器上的消息处理负载可能并不高,可以用一个consumer来处理其消息, 当然为NORMAL_EXCHANGE绑定多个队列或者运行多个consumer也是可以的. 使用死信交换器时, 其架构如下(使用一个worker consumer队列的情况):

首先, 将该类导出为Runnable JAR包.打开一个控制台终端,运行一个worker consumer:

java -jar deadletterexchange.jar consumer

打开另一个控制台终端, 运行一个DLX consumer:

java -jar deadletterexchange.jar dlxconsumer

打开另一个控制台终端, 运行一个producer:

java -jar deadletterexchange.jar producer

producer发送完消息后, worker consumer的log输出为:

Received normal exchange msg:[normal message.] msg id:0 reject:false
Received normal exchange msg:[normal message.] msg id:1 reject:true
Received normal exchange msg:[normal message.] msg id:2 reject:false
Received normal exchange msg:[normal message.] msg id:4 reject:false
Received normal exchange msg:[normal message.] msg id:5 reject:true
Received normal exchange msg:[normal message.] msg id:6 reject:false
Received normal exchange msg:[normal message.] msg id:8 reject:false
Received normal exchange msg:[normal message.] msg id:9 reject:true
Received normal exchange msg:[normal message.] msg id:10 reject:false
Received normal exchange msg:[normal message.] msg id:12 reject:false
Received normal exchange msg:[normal message.] msg id:13 reject:true
Received normal exchange msg:[normal message.] msg id:14 reject:false
Received normal exchange msg:[normal message.] msg id:16 reject:false
Received normal exchange msg:[normal message.] msg id:17 reject:true
Received normal exchange msg:[normal message.] msg id:18 reject:false

DLX consumer的log输出为:

Received dead letter exchange msg:[normal message.] msg id:1
Received dead letter exchange msg:[normal message.] msg id:5
Received dead letter exchange msg:[normal message.] msg id:9
Received dead letter exchange msg:[normal message.] msg id:13
Received dead letter exchange msg:[normal message.] msg id:17

在worker consumer中, 对部分消息使用了basicReject进行拒绝回复, 并且其requeue属性为false. 从worker consumer的log输出看,这些被拒绝的消息ID为:1,5,9,13,17. 再来看DLX consumer的log输出,发现worker consumer上这些被拒绝回复的消息全部路由到了DLX交换器上了.

对于死信交换器需要注意的几个细节是:

• 死信交换器相较于非死信交换器没有任何特殊的属性, 可以为其指定为任何交换器类型(例子中使用了direct类型).唯一要做的是在定义worker consumer的队列时, 使用参数x-dead-letter-exchange指定死信交换器即可.
• 在worker consumer中定义队列指定死信交换器时, 该死信交换器甚至可以不存在(即还没有定义). 示例中先在终端中运行了worker consumer,然后运行了DLX consumer. 在运行worker consumer时, 死信交换器还没有被定义.但是当有消息需要被路由到死信交换器时, 其必须已经存在.
• worker consumer可以使用basicReject或者basicNack对消息进行确认, 当这两个命令的requeue参数设置为false时, 该消息才被路由到死信交换器.否则, 消息将被重新在worker consumer间进行投递. 需要注意的是, 重新投递的消息可能还会被投递到拒绝该消息的worker consumer上, 形成循环.
• 如果producer发送消息时, 如果该消息无法路由到任何队列, 那么这个消息是不会被路由到死信交换器的. 示例中producer发送的部分消息,其route key为not_existed_route_key, 这些消息无法路由到任何队列, 也不会被路由到死信交换器.
• 默认情况下, 路由到死信交换器的消息,将使用producer发送时的route key. 但是, 可以改变路由到死信交换器的route key. 在worker consumer定义其队列时, 除了使用x-dead-letter-exchange参数指定死信交换器, 还可以使用x-dead-letter-routing-key参数指定该消息的route key.使用这个配置可以在DLX worker中判断出消息是从哪个worker consumer路由过来的.

BTW: 因为这个示例导出的Runnable JAR包中包含了RabbitMQ的Java客户端库,所以可以直接运行. 否则,运行时需要在classpath中指定其客户端库路径.