Rabbimq的作用,解耦,消峰,异步通信等。
a. rabbitmq发送消息,如果不进行特殊设置(设置为事务模式或者确认模型),则生产者可靠性无法保证,默认rabbitmq不会返回任何信息给生产者。
b.事务模式的缺点,事务模式本身为同步机制,性能上比确认模式要差,确认模式为异步模式(与mqserver连接的信道也为双向信道)。
package com.rabbitmq.test;
import com.rabbitmq.client.*;
import java.io.IOException;
import java.util.concurrent.TimeoutException;
public class ProducerV5 {
static String EXCHANGE_NAME = "exchange_name";
static String QUEUE_NAME = "queue_name";
static String ROUTING_KEY = "root_key";
public static void main(String[] args) throws Exception{
ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("****");
factory.setPassword("****");
factory.setVirtualHost("/");
factory.setHost("**.**.**.**");
factory.setPort(5672);
Connection conn = null;
final Channel channel;
try {
conn = factory.newConnection();
//注意conn不关闭时,程序不停止。
channel = conn.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME,"direct", true, false, null);
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, ROUTING_KEY);
//当rabbimq无法将消息正确路由到队列时,触发。
channel.addReturnListener(new ReturnListener() {
public void handleReturn(int replyCode, String replyText, String exchange, String routingKey, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("replyCode:"+replyCode);
System.out.println("replyText:"+replyText);
System.out.println("exchange:"+exchange);
System.out.println("routingKey:"+routingKey);
String string = new String(body);
System.out.println("basic return"+string);
}
});
int i=1;
while (i<20){
String string = "hello world"+i;
//rabbitmq无法找到EXCHANGE_NAME时,会触发异常。无法找到queue时,mq会发送Basic.Return命令给生产者,addReturnListener进行监听。
channel.basicPublish(EXCHANGE_NAME, ROUTING_KEY+"uuuu", true, MessageProperties.PERSISTENT_TEXT_PLAIN,
string.getBytes());
i++;
System.out.println(i+"ddd");
}
System.out.println("dddfdafdf");
//channel.addReturnListener();
} catch (IOException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}finally {
conn.close();
}
System.out.println("end");
}
}
package com.rabbitmq.test;
import com.rabbitmq.client.*;
import org.omg.CosNaming.NamingContextExtPackage.StringNameHelper;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeoutException;
public class ProducerV5 {
static String EXCHANGE_NAME = "exchange_name_v2";
static String BEI_FEN_EXCHANGE_NAME = "bei_fen_exchange_name";
static String UN_ROUTED_QUEUE_NAME = "un_routed_exchange_name";
static String QUEUE_NAME = "queue_name";
static String ROUTING_KEY = "root_key";
public static void main(String[] args) throws Exception{
ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("***");
factory.setPassword("***");
factory.setVirtualHost("/");
factory.setHost("**.**.**.**");
factory.setPort(5672);
Connection conn = null;
final Channel channel;
try {
conn = factory.newConnection();
channel = conn.createChannel();
Map<String,Object> argsProperties = new HashMap<String, Object>();
argsProperties.put("alternate-exchange", BEI_FEN_EXCHANGE_NAME);
channel.exchangeDeclare(EXCHANGE_NAME,"direct", true, false, argsProperties);
channel.exchangeDeclare(BEI_FEN_EXCHANGE_NAME,"fanout", true, false, null);
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
channel.queueDeclare(UN_ROUTED_QUEUE_NAME, true, false, false, null);
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, ROUTING_KEY);
channel.queueBind(UN_ROUTED_QUEUE_NAME, BEI_FEN_EXCHANGE_NAME, "");
int i=1;
while (i<20){
String string = "hello world"+i;
//此时路由不到队列时,消息发送到备份交换机。
channel.basicPublish(EXCHANGE_NAME, ROUTING_KEY+"uuuufff", true, MessageProperties.PERSISTENT_TEXT_PLAIN,
string.getBytes());
i++;
System.out.println(i+"ddd");
}
//channel.addReturnListener();
} catch (IOException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}finally {
conn.close();
}
System.out.println("end");
}
}
注意:备份交换机与普通交换机没有区别,发送到备份交换机的路由key与正常交换机一样。所以备份交换机建议设置为fanout。
package com.rabbitmq.test;
import com.rabbitmq.client.*;
import java.io.IOException;
import java.util.TreeMap;
import java.util.concurrent.TimeoutException;
public class ProducerV3 {
static String EXCHANGE_NAME = "exchange_name";
static String QUEUE_NAME = "queue_name";
static String ROUTING_KEY = "root_key";
public static void main(String[] args) {
ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("***");
factory.setPassword("****");
factory.setVirtualHost("/");
factory.setHost("10.12.24.104");
factory.setPort(5672);
Connection conn = null;
final Channel channel;
final TreeMap<Long, String> map = new TreeMap<Long, String>();
try {
conn = factory.newConnection();
channel = conn.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME,"direct", true, false, null);
channel.exchangeDeclare(EXCHANGE_NAME+"dddd","direct", true, false, null);
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, ROUTING_KEY);
//设置mq为认证模式。
channel.confirmSelect();
channel.addReturnListener(new ReturnListener() {
public void handleReturn(int replyCode, String replyText, String exchange, String routingKey, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("replyCode:"+replyCode);
System.out.println("replyText:"+replyText);
System.out.println("exchange:"+exchange);
System.out.println("routingKey:"+routingKey);
String string = new String(body);
System.out.println("basic return"+string);
}
});
//rabbitmq发送Basic.Ack.(这里测试与书上说的不一样,并不是正确的投递到队列,才会发送Basic.Ack,自己测试得到的结果消息到达了交换机就会发送Ack,所以发送认证机制最好跟备份交换机或者madatory参数共同保证可靠性。Nack理解可能是rabbitmq因为自身错误,无法接收消息时,才发送Nack。)
channel.addConfirmListener(new ConfirmListener() {
public void handleAck(long deliveryTag, boolean multiple) throws IOException {
System.out.println("handleAck"+deliveryTag);
map.remove(deliveryTag);
}
//mutiple为之前的消息是否正确投递
public void handleNack(long deliveryTag, boolean multiple) throws IOException {
System.out.println("handleNack"+deliveryTag);
// 重新发送一条相同的消息。注意信道认证模式时,没发送到信道的消息都有唯一的deliveryTag.
String string = map.get(deliveryTag);
channel.basicPublish(EXCHANGE_NAME, ROUTING_KEY, true, MessageProperties.PERSISTENT_TEXT_PLAIN,
string.getBytes());
map.clear(deliveryTag);
}
});
int i=1;
while (i<20){
String string = "hello world"+i;
long nextNo = channel.getNextPublishSeqNo();
//这里无法路由到正确的队里,仍然会触发ack.
channel.basicPublish(EXCHANGE_NAME, ROUTING_KEY+"gggggg", true, MessageProperties.PERSISTENT_TEXT_PLAIN,
string.getBytes());
i++;
map.put(nextNo, string);
}
System.out.println("dddfdafdf");
//channel.addReturnListener();
} catch (IOException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}
}
}
package com.rabbitmq.test;
import com.rabbitmq.client.*;
import java.io.IOException;
import java.util.ArrayList;
import java.util.concurrent.TimeoutException;
public class ProducerV4 {
static String EXCHANGE_NAME = "exchange_name";
static String QUEUE_NAME = "queue_name";
static String ROUTING_KEY = "root_key";
public static void main(String[] args) {
ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("kreditplus_dev");
factory.setPassword("kreditplus_dev");
factory.setVirtualHost("/");
factory.setHost("10.12.24.104");
factory.setPort(5672);
Connection conn = null;
Channel channel = null;
try {
conn = factory.newConnection();
channel = conn.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME,"direct", true, false, null);
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, ROUTING_KEY);
channel.confirmSelect();
ArrayList<String> arraylist = new ArrayList<String>();
channel.addReturnListener(new ReturnListener() {
public void handleReturn(int replyCode, String replyText, String exchange, String routingKey, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("replyCode:" + replyCode);
System.out.println("replyText:" + replyText);
System.out.println("exchange:" + exchange);
System.out.println("routingKey:" + routingKey);
String string = new String(body);
System.out.println("basic return" + string);
}
});
int i=1;
int mod = 0;
while (i<40) {
String string = "hello world" + i;
long nextNo = channel.getNextPublishSeqNo();
System.out.println("nextNo"+ nextNo);
channel.basicPublish(EXCHANGE_NAME, ROUTING_KEY, true, MessageProperties.PERSISTENT_TEXT_PLAIN,
string.getBytes());
i++;
arraylist.add(string);
if (++mod >= 20) {
mod = 0;
try {
//批量认证模式
boolean value = channel.waitForConfirms();
if (!value) {
//重复发送
for (String string_v2 : arraylist) {
channel.basicPublish(EXCHANGE_NAME, "dadfa", true, MessageProperties.PERSISTENT_TEXT_PLAIN,
string_v2.getBytes());
}
System.out.println();
} else {
arraylist.clear();
System.out.println("success" + string);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
} catch (IOException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}
}
}
交换机持久化:保证重启时,交换机元数据不丢失。不持久化,则mq重启后,交换机无法继续使用。 不影响消息是否丢失。
队列持久化:队列不持久化时,重启后消息丢失。
消息持久化:发送消息时,deliveryMode设置为2,将消息持久化,如果只有队列持久化,没有消息持久化,重启后,消息会丢失。
消息持久化到磁盘时,可能需要在操作系统缓存保存一段时间,才能够存到物理磁盘中,这段时间如果rabbitmq挂掉,可能会丢失数据,可通过镜像队里进行保证。
消息保存时间有两个限制:
1.消息本身的的保存时间TTL和队列的保存时间TTL,消息的过期时间,取两者中小的那个,不设置TTL,则消息会等到消费后,才会从rabbitmq删除。
1.消费者可靠性保证主要利用手动确认机制。
2.如果多个消费者订阅同一队列,rabbitmq采用轮训方式推送消息到消费者。
3. channel.basicQos(2),控制了单个消费者,在信道上未确认的个数,channel.basixQos(10, true), global为true整体信道上的消费者都需要遵从这个限定值,(加和限定和单个限定有待实验)。
package com.rabbitmq.test;
import com.rabbitmq.client.*;
import java.io.IOException;
import java.util.concurrent.TimeoutException;
public class ConsumerTestV2 {
static String QUEUE_NAME = "queue_name";
public static void main(String[] args) {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("10.12.24.104");
factory.setPort(5672);
factory.setUsername("kreditplus_dev");
factory.setPassword("kreditplus_dev");
//factory.setVirtualHost("/");
Connection connection = null;
try {
connection = factory.newConnection();
final Channel channel = connection.createChannel();
//设置客户端最多接收未被ack的消息的个数, 起到了一个滑动窗口的作用,channel.basicQos还有针对一个消费者和多个消费者的区别。
channel.basicQos(2);
System.out.println("aaaaaa");
boolean autoAck = false;
channel.basicConsume(QUEUE_NAME, autoAck, "myConsumerTagV2",
new DefaultConsumer(channel) {
@Override
public void handleDelivery(String consumerTag,
Envelope envelope,
AMQP.BasicProperties properties,
byte[] body)
throws IOException
{
String routingKey = envelope.getRoutingKey();
String contentType = properties.getContentType();
long deliveryTag = envelope.getDeliveryTag();
// (process the message components here ...)
System.out.print("aaaaaavdadfadsa"+new String(body)+deliveryTag);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
channel.basicAck(deliveryTag, false);
}
});
// channel.close();
// connection.close();
} catch (IOException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}finally {
}
}
}
消息在一个队列中变成死信,能够过重新被发送到一个交换机DLX,
绑定DLX的队列就称之为死信队列。
消息变成死信的情况:
1.消息被拒绝
2.消息过期
3.队列达到最大长度。
生产者
package com.rabbitmq.test;
import com.rabbitmq.client.*;
import com.rabbitmq.client.AMQP.BasicProperties;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeoutException;
public class Producer {
static String EXCHANGE_NAME = "exchange_name";
static String QUEUE_NAME = "queue_name_v2";
static String ROUTING_KEY = "root_key";
static String DEAD_DLX_EXCHANGE = "dlx_exchange";
static String DEAD_QUEUE = "dlx_queue";
public static void main(String[] args) {
ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("kreditplus_dev");
factory.setPassword("kreditplus_dev");
factory.setVirtualHost("/");
factory.setHost("10.12.24.104");
factory.setPort(5672);
Connection conn = null;
Channel channel = null;
try {
conn = factory.newConnection();
channel = conn.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME,"direct", true, false, null);
//dead exchange
channel.exchangeDeclare(DEAD_DLX_EXCHANGE,"direct", true, false, null);
Map<String,Object> ars= new HashMap<String,Object>();
ars.put("x-dead-letter-exchange", DEAD_DLX_EXCHANGE);
//也可以为这个DLX指定路由键,如果没有特殊指定,则使用原队列的路由键
ars.put("x-dead-letter-routing-key","dlx-routing-key");
//给正常队列绑定死信交换机
channel.queueDeclare(QUEUE_NAME, true, false, false, ars);
channel.queueDeclare(DEAD_QUEUE, true, false, false, null);
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, ROUTING_KEY);
//死信交换机、死信队列、路由key绑定
channel.queueBind(DEAD_QUEUE, DEAD_DLX_EXCHANGE, "dlx-routing-key");
int i=10;
while (i<20){
String string = "hello world"+i;
channel.basicPublish(EXCHANGE_NAME, ROUTING_KEY, new BasicProperties.Builder().deliveryMode(2).build(), //设置为持久化
string.getBytes());
i++;
}
channel.close();
conn.close();
} catch (IOException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}
System.out.println("aaaaaa");
}
}
消费者
package com.rabbitmq.test;
import com.rabbitmq.client.*;
import java.io.IOException;
import java.util.concurrent.TimeoutException;
public class ConsumerTest {
static String QUEUE_NAME = "queue_name";
public static void main(String[] args) {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("10.12.24.104");
factory.setPort(5672);
factory.setUsername("kreditplus_dev");
factory.setPassword("kreditplus_dev");
//factory.setVirtualHost("/");
Connection connection = null;
try {
connection = factory.newConnection();
final Channel channel = connection.createChannel();
//设置客户端最多接收未被ack的消息的个数
channel.basicQos(3);
System.out.println("aaaaaa");
boolean autoAck = false;
channel.basicConsume(QUEUE_NAME, autoAck, "myConsumerTag",
new DefaultConsumer(channel) {
@Override
public void handleDelivery(String consumerTag,
Envelope envelope,
AMQP.BasicProperties properties,
byte[] body)
throws IOException
{
String routingKey = envelope.getRoutingKey();
String contentType = properties.getContentType();
long deliveryTag = envelope.getDeliveryTag();
// (process the message components here ...)
System.out.println("aaaaaavdadfadsa"+new String(body));
System.out.println(deliveryTag);
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
e.printStackTrace();
}
//重点是这里最后一个参数决定了是否让消息重新回到队里,还是转发到死信队列中。
channel.basicNack(deliveryTag, false, false);
//channel.basicAck(deliveryTag, false);
}
});
// channel.close();
// connection.close();
} catch (IOException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}finally {
}
}
}
延迟队列是在死信队列的基础上实现的,可通过消息的TTL来控制,生产者消息生产时,设置死信队列和消息TTL时间,消息过期进入到死信队列。 消费者消费死信队列保证了延迟功能。
生产者
package com.rabbitmq.test;
import com.rabbitmq.client.*;
import com.rabbitmq.redis.RedisPool;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import java.io.IOException;
import java.util.TreeMap;
import java.util.UUID;
import java.util.concurrent.TimeoutException;
public class ProducerV6 {
static String EXCHANGE_NAME = "exchange_name";
static String QUEUE_NAME = "queue_name";
static String ROUTING_KEY = "root_key";
public static void main(String[] args) {
ConnectionFactory factory = new ConnectionFactory();
factory.setUsername("admin");
factory.setPassword("admin");
factory.setVirtualHost("/");
factory.setHost("192.168.0.17");
factory.setPort(5672);
Connection conn = null;
final Channel channel;
final TreeMap<Long, String> map = new TreeMap<Long, String>();
try {
conn = factory.newConnection();
channel = conn.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME,"direct", true, false, null);
channel.exchangeDeclare(EXCHANGE_NAME+"dddd","direct", true, false, null);
channel.queueDeclare(QUEUE_NAME, true, false, false, null);
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, ROUTING_KEY);
channel.confirmSelect();
channel.addReturnListener(new ReturnListener() {
public void handleReturn(int replyCode, String replyText, String exchange, String routingKey, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("replyCode:"+replyCode);
System.out.println("replyText:"+replyText);
System.out.println("exchange:"+exchange);
System.out.println("routingKey:"+routingKey);
String string = new String(body);
System.out.println("basic return"+string);
}
});
channel.addConfirmListener(new ConfirmListener() {
public void handleAck(long deliveryTag, boolean multiple) throws IOException {
System.out.println("handleAck"+deliveryTag);
map.remove(deliveryTag);
}
public void handleNack(long deliveryTag, boolean multiple) throws IOException {
System.out.println("handleNAck"+deliveryTag);
map.remove(deliveryTag);
// string need has primary key,
// repeat send
String string = map.get(deliveryTag);
channel.basicPublish(EXCHANGE_NAME, ROUTING_KEY, true, MessageProperties.PERSISTENT_TEXT_PLAIN,
string.getBytes());
}
});
//这里为了做实验保证幂等性,实际corrId每条消息唯一。
String corrId = "10389234451254177779900";
AMQP.BasicProperties props = new AMQP.BasicProperties.Builder()
.deliveryMode(2)
.correlationId(corrId)
.build();
int i=1;
while (i<20){
String string = "hello world"+i;
long nextNo = channel.getNextPublishSeqNo();
channel.basicPublish(EXCHANGE_NAME, ROUTING_KEY, false, props,
string.getBytes());
i++;
map.put(nextNo, string);
}
System.out.println("dddfdafdf");
//channel.addReturnListener();
} catch (IOException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}
}
}
消费者
消息重复发送的原因有,重试,生产者重复发送等,需要自己在消费者端保证消息的唯一性。
方法有
1.利用redis的setnx命令判断是否已经消费过,缓存时间的设置,要看内存的大小,及消息重复发送的时间间隔进行评估。
2.利用mysql的唯一健,但是高并发的情况下,对是数据库压力较大。
package com.rabbitmq.test;
import com.rabbitmq.client.*;
import com.rabbitmq.redis.RedisPool;
import redis.clients.jedis.Jedis;
import java.io.IOException;
import java.util.concurrent.TimeoutException;
public class ConsumerTestV3 {
static String QUEUE_NAME = "queue_name";
static Jedis jedis = RedisPool.getJedis();
public static void main(String[] args) {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("192.168.0.17");
factory.setPort(5672);
factory.setUsername("admin");
factory.setPassword("admin");
//factory.setVirtualHost("/");
Connection connection = null;
try {
connection = factory.newConnection();
final Channel channel = connection.createChannel();
//设置客户端最多接收未被ack的消息的个数
channel.basicQos(3);
System.out.println("aaaaaa");
boolean autoAck = false;
channel.basicConsume(QUEUE_NAME, autoAck, "myConsumerTag",
new DefaultConsumer(channel) {
@Override
public void handleDelivery(String consumerTag,
Envelope envelope,
AMQP.BasicProperties properties,
byte[] body)
throws IOException
{
String routingKey = envelope.getRoutingKey();
String contentType = properties.getContentType();
long deliveryTag = envelope.getDeliveryTag();
String uuid = properties.getCorrelationId();
System.out.println(uuid);
//缓存超时时间需要根据业务指定
if(jedis.set(uuid,"success","NX","PX",30000)!=null){
// (process the message components here ...)
System.out.println(new String(body));
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
channel.basicNack(deliveryTag, false, false);
//channel.basicAck(deliveryTag, false);
}
});
// channel.close();
// connection.close();
} catch (IOException e) {
e.printStackTrace();
} catch (TimeoutException e) {
e.printStackTrace();
}finally {
}
}
}
rabbitmq的顺序性实现的话核心原理
1.按顺序的进入队列
2.按顺序的处理消息
3.重试异常一起的消息重试如何保证顺序。
1.按顺序进入队列需要保证,生产者唯一,发送消息时保证顺序,比如数据1,数据2,数据3,按顺序发送到同一队列。
2.消费者处理消息时,处理数据1,数据2,数据3时,核心是保证一个消费者线程处理数据1,数据2,数据3.
3.发送超时重试等现象数据被破坏,个人理解可通过状态机来保证。存储每个数据的三种状态。
这篇文章:https://blog.csdn.net/weixin_40816738/article/details/105704335
a. 队列元数据:队列名称和它的属性
b. 交换器元数据:交换器名称、类型和属性
c. 绑定元数据:一张简单的表格展示了如何将消息路由到队列
d. vhost元数据:为vhost内的队列、交换器和绑定提供命名空间和安全属性
第一,存储空间。如果每个集群节点都拥有所有Queue的完全数据拷贝,那么每个节点的存储空间会非常大,集群的消息积压能力会非常弱(无法通过集群节点的扩容提高消息积压能力);
第二,性能。消息的发布者需要将消息复制到每一个集群节点,对于持久化消息,网络和磁盘同步复制的开销都会明显增加。
a. 如果有一个消息生产者或者消息消费者通过amqp-client的客户端连接至节点1进行消息的发布或者订阅,那么此时的集群中的消息收发只与节点1相关。
b.客户端连接的是非队列数据所在节点
非数据所在节点只会起到一个路由转发的作用,最终负责存储和消费的消息还是会在队列数据真正所在的节点上。
磁盘节点:持久化存储原数据,如果是单点部署,那么一定是磁盘节点,如果磁盘节点挂掉,则不能创建队列,交换器等。
内存节点:将配置信息和原数据信息保存在内存中,通常负责整个集群与客户端的连接。
下图未将磁盘节点和内存节点进行区分,可理解为节点均需持久化原数据。
keepalived机制利用(虚拟路由冗余协议),已软件的形式实现服务的热备功能,(通常是两台linux服务器组成热备组,master和backup,同一时间只有master对外服务,master会虚拟出一个ip地址,简称VIP,这个VIP只存在于master上对外服务。如果keepalived检测到master故障,备份服务器backup自动接管VIP并成为master. keepalived将原master移除,当源master恢复后,会自动加入到热备组,默认再抢占,成为master。
镜像队列是基于普通的集群模式的,然后再添加一些策略,所以还是得先配置普通集群,然后才能设置镜像队列。镜像队列存在于多个节点。要实现镜像模式,需要先搭建一个普通集群模式,在这个模式的基础上再配置镜像模式以实现高可用。
镜像队列结构如下:
所有对mirror_queue_master的操作,会通过可靠组播GM的方式同步到各slave节点。
GM负责消息的广播,mirror_queue_slave负责回调处理,而master上的回调处理是由coordinator负责完成。mirror_queue_slave中包含了普通的BackingQueue进行消息的存储,master节点中BackingQueue包含在mirror_queue_master中由AMQQueue进行调用。
整体流程个人理解:mirror_queue_master当收到操作时,触发master对应的GM,GM收到消息并进行回调Coordinator. 同时AMQQueue也会调用BackingQueue进行master消息的持久化,
master的GM通过链表的方式传递消息,当slave的GM收到消息时,回调mirror_queue_slave进行处理,将消息持久化到或镜像队列所在BackingQueue。 此时可保证master节点在持久化数据时,数据还未刷到磁盘中,挂掉了,可通过镜像队列保证可靠性。
GM模块实现的一种可靠的组播通讯协议,该协议能够保证组播消息的原子性,即保证组中活着的节点要么都收到消息要么都收不到。
它的实现大致如下:
将所有的节点形成一个循环链表,每个节点都会监控位于自己左右两边的节点,当有节点新增时,相邻的节点保证当前广播的消息会复制到新的节点上;当有节点失效时,相邻的节点会接管保证本次广播的消息会复制到所有的节点。在master节点和slave节点上的这些gm形成一个group,group(gm_group)的信息会记录在mnesia中。不同的镜像队列形成不同的group。消息从master节点对于的gm发出后,顺着链表依次传送到所有的节点,由于所有节点组成一个循环链表,master节点对应的gm最终会收到自己发送的消息,这个时候master节点就知道消息已经复制到所有的slave节点了。
新增节点:
每当一个节点加入或者重新加入(例如从网络分区中恢复过来)镜像队列,之前保存的队列内容会被清空。
slave节点失效,系统处理做些记录外几乎啥都不做。
master节点失效:
1.与客户端连接全部断开
2.选取最老的slave节点作为master, 如果此时slave为同步master数据,则数据丢失。
3.新的master重新入队列所有为没有ack的消息,此时客户端可能会有重复消息。