背景
- 进程间通讯和系统间的消息通知,比如在分布式系统中。
- 解耦,比如像我们公司有许多开发团队,每个团队负责业务的不同模块,各个开发团队可以使用MQ来通信。
- 在一些高并发场景下,使用MQ的异步特性。
知识点介绍(copy)
1.MQ
为什么使用MQ
(1)解耦
在项目启动之初来预测将来项目会碰到什么需求,是极其困难的。消息系统在处理过程中间插入了一个隐含的、基于数据的接口层,两边的处理过程都要实现这一接口。这允许你独立的扩展或修改两边的处理过程,只要确保它们遵守同样的接口约束。
(2)冗余
有些情况下,处理数据的过程会失败。除非数据被持久化,否则将造成丢失。消息队列把数据进行持久化直到它们已经被完全处理,通过这一方式规避了数据丢失风险。许多消息队列所采用的”插入-获取-删除”范式中,在把一个消息从队列中删除之前,需要你的处理系统明确的指出该消息已经被处理完毕,从而确保你的数据被安全的保存直到你使用完毕。
(3)扩展性
因为消息队列解耦了你的处理过程,所以增大消息入队和处理的频率是很容易的,只要另外增加处理过程即可。不需要改变代码、不需要调节参数。扩展就像调大电力按钮一样简单。
(4)灵活性 & 峰值处理能力
在访问量剧增的情况下,应用仍然需要继续发挥作用,但是这样的突发流量并不常见;如果为以能处理这类峰值访问为标准来投入资源随时待命无疑是巨大的浪费。使用消息队列能够使关键组件顶住突发的访问压力,而不会因为突发的超负荷的请求而完全崩溃。
(5)可恢复性
系统的一部分组件失效时,不会影响到整个系统。消息队列降低了进程间的耦合度,所以即使一个处理消息的进程挂掉,加入队列中的消息仍然可以在系统恢复后被处理。
(6)顺序保证
在大多使用场景下,数据处理的顺序都很重要。大部分消息队列本来就是排序的,并且能保证数据会按照特定的顺序来处理。Kafka保证一个Partition内的消息的有序性。
(7)缓冲
在任何重要的系统中,都会有需要不同的处理时间的元素。例如,加载一张图片比应用过滤器花费更少的时间。消息队列通过一个缓冲层来帮助任务最高效率的执行———写入队列的处理会尽可能的快速。该缓冲有助于控制和优化数据流经过系统的速度。
(8)异步通信
很多时候,用户不想也不需要立即处理消息。消息队列提供了异步处理机制,允许用户把一个消息放入队列,但并不立即处理它。想向队列中放入多少消息就放多少,然后在需要的时候再去处理它们。
MQ的特点
(1)应用解耦
消息是与平台和语言无关的,消息队列可以应对多变的产品变更。
(2)异步通信
可以缩短请求等待的时间,使用专门处理请求的消费者来执行,提高WEB页面的吞吐量,
尤其是瞬间发生的高流量情况,消息队列非常有助于顶住访问的压力
(3)数据持久化
未完成的消息不会因为某些故障而丢失。
(4)送达保证
消息队列提供的冗余机制保证了消息确实能被处理,除非消费者明确表示已经出来完这个消息,否则这个消息可以被放回队列中以备其他消费者出来。
2.amqp
AMQP,即Advanced Message Queuing Portocol,高级消息队列协议。
该协议使得遵从该规范的客户端应用和消息中间服务器的全功能互操作成为可能,
它的设计初衷是为了拜托商业MQ高额费用和不同MQ供应商的接口不统一的问题。
消息发送与接收的双方遵守这个协议可以实现异步通信,这个协议约定了消息的格式和工作方式
为什么需要amqp
在分布式的系统中,子系统如果使用socket进行通信,有很多问题需要解决
1)消息的发送者和接收者如何维持这个连接,如果一方中断,这期间的数据如何防止丢失?
2)如何降低发送者和接收者的耦合
3)如何让优先级高的接收者接收消息
4)如何做到负载均衡
5)如何将消息发送给相关的接收者,如果接收者订阅了不同的数据,如何正确的分发到接受者。
6)如何保证接收者接到了正确的或者是有序的数据。
7)如何做到可扩展,将通信模块发到集群上去。
AMQP可以解决这些问题。
项目实例
pom.xml
org.springframework.cloud
spring-cloud-starter-bus-amqp
1.接口封装
(1)消息队列走getChatRequest()方法,调用方传参的vo需要继承RequestIM;
/**
* @Auther: Young
* @Date: 2018/9/1 16:27
* @Description:
*/
@Service
public class RequestIMServiceImpl implements RequestIMService {
private static final Logger logger = LoggerFactory.getLogger(RequestIMServiceImpl.class);
@Value("${im.appKey}")
private String appKey;
@Value("${im.appSecret}")
private String appSecret;
@Autowired
private RabbitService rabbitService;
/**
*
* 功能描述: 接口不需要实时返回值,走消息队列
*
* @auther: Young
* @date: 2018/9/1 16:27
*/
@Override
public void getChatRequest(String url, RequestIM vo) {
Map map = new HashMap<>();
map.put(ChatConstant.MESSAGE_PARAMETER.URL,url);
map.put(ChatConstant.MESSAGE_PARAMETER.VO,vo);
MQMessage message = new MQMessage(ChatConstant.MESSAGE_PARAMETER.ENENT, map);
//Direct Exchange单播、
//Topic Exchange组播、
//Fanout Exchange广播
rabbitService.send(Excharge.direct, ChatConstant.MESSAGE_PARAMETER.ROUTING_KEY, message);
}
/**
*
* 功能描述: 接口需要实时返回值,走普通http
*
* @auther: Young
* @date: 2018/9/1 16:27
*/
@Override
public T getChatRequestWithReturn(String url, RequestIM vo, Class T) {
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost(url);
String nonce = StringUtil.createRandom(false, 20);
String curTime = String.valueOf((new Date()).getTime() / 1000L);
String checkSum = ImCheckSum.getCheckSum(appSecret, nonce, curTime);//参考 计算CheckSum的java代码
try {
// 设置请求的header
httpPost.addHeader("AppKey", appKey);
httpPost.addHeader("Nonce", nonce);
httpPost.addHeader("CurTime", curTime);
httpPost.addHeader("CheckSum", checkSum);
httpPost.addHeader("Content-Type", "application/x-www-form-urlencoded;charset=utf-8");
Map map = new HashMap();
List nvps = new ArrayList();
map = StringUtil.beanToMap(vo);
Iterator> entries = map.entrySet().iterator();
while (entries.hasNext()) {
Map.Entry entry = entries.next();
if (null == entry.getValue()) {
nvps.add(new BasicNameValuePair(entry.getKey(), null));
} else if (entry.getValue() instanceof String) {
nvps.add(new BasicNameValuePair(entry.getKey(), entry.getValue().toString()));
} else {
nvps.add(new BasicNameValuePair(entry.getKey(), JSONUtils.toJSONString(entry.getValue())));
}
}
httpPost.setEntity(new UrlEncodedFormEntity(nvps, "utf-8"));
logger.info(url + new UrlEncodedFormEntity(nvps, "utf-8"));
// 执行请求
HttpResponse response = httpClient.execute(httpPost);
//.entity所得到的流是不可重复读取的也就是说所得的到实体只能一次消耗完,
//不能多次读取,所以在执行html = EntityUtils.toString(entity)后,流就关闭了,就导致后面的读和写显示错误.
String responseString = EntityUtils.toString(response.getEntity());
logger.info("==========:" + responseString + "===========");
return JsonUtil.fromJson(responseString, T);
} catch (Exception e) {
throw new LFException(GroupCode.request_im_error);
}
}
}
2,MQMessage.java
public class MQMessage implements Serializable {
private static final long serialVersionUID = 1L;
private String id;
private Long expire;
private String event;
private Object data;
private Integer sendTimes = 1;
public MQMessage(String event, Object data) {
this.event = event;
this.data = data;
}
public MQMessage() {
}
public String getId() {
return this.id;
}
public Long getExpire() {
return this.expire;
}
public String getEvent() {
return this.event;
}
public Object getData() {
return this.data;
}
public Integer getSendTimes() {
return this.sendTimes;
}
public void setId(final String id) {
this.id = id;
}
public void setExpire(final Long expire) {
this.expire = expire;
}
public void setEvent(final String event) {
this.event = event;
}
public void setData(final Object data) {
this.data = data;
}
public void setSendTimes(final Integer sendTimes) {
this.sendTimes = sendTimes;
}
public boolean equals(final Object o) {
if (o == this) {
return true;
} else if (!(o instanceof MQMessage)) {
return false;
} else {
MQMessage other = (MQMessage)o;
if (!other.canEqual(this)) {
return false;
} else {
label71: {
Object this$id = this.getId();
Object other$id = other.getId();
if (this$id == null) {
if (other$id == null) {
break label71;
}
} else if (this$id.equals(other$id)) {
break label71;
}
return false;
}
Object this$expire = this.getExpire();
Object other$expire = other.getExpire();
if (this$expire == null) {
if (other$expire != null) {
return false;
}
} else if (!this$expire.equals(other$expire)) {
return false;
}
label57: {
Object this$event = this.getEvent();
Object other$event = other.getEvent();
if (this$event == null) {
if (other$event == null) {
break label57;
}
} else if (this$event.equals(other$event)) {
break label57;
}
return false;
}
Object this$data = this.getData();
Object other$data = other.getData();
if (this$data == null) {
if (other$data != null) {
return false;
}
} else if (!this$data.equals(other$data)) {
return false;
}
Object this$sendTimes = this.getSendTimes();
Object other$sendTimes = other.getSendTimes();
if (this$sendTimes == null) {
if (other$sendTimes == null) {
return true;
}
} else if (this$sendTimes.equals(other$sendTimes)) {
return true;
}
return false;
}
}
}
protected boolean canEqual(final Object other) {
return other instanceof MQMessage;
}
public int hashCode() {
int PRIME = true;
int result = 1;
Object $id = this.getId();
int result = result * 59 + ($id == null ? 43 : $id.hashCode());
Object $expire = this.getExpire();
result = result * 59 + ($expire == null ? 43 : $expire.hashCode());
Object $event = this.getEvent();
result = result * 59 + ($event == null ? 43 : $event.hashCode());
Object $data = this.getData();
result = result * 59 + ($data == null ? 43 : $data.hashCode());
Object $sendTimes = this.getSendTimes();
result = result * 59 + ($sendTimes == null ? 43 : $sendTimes.hashCode());
return result;
}
public String toString() {
return "MQMessage(id=" + this.getId() + ", expire=" + this.getExpire() + ", event=" + this.getEvent() + ", data=" + this.getData() + ", sendTimes=" + this.getSendTimes() + ")";
}
}
3.RabbitService.java
@Component
public class RabbitService {
private static final Logger log = LoggerFactory.getLogger(RabbitService.class);
@Autowired
private RabbitTemplate template;
@Autowired
private MQSendCache sendCache;
public RabbitService() {
}
public void send(Excharge excharge, String routingKey, MQMessage message) {
this.send(excharge.toString(), routingKey, message);
}
public void send(String excharge, String routingKey, MQMessage message) {
CorrelationData correlationData = new CorrelationData(UUIDUtil.random());
message.setId(correlationData.getId());
RabbitService.LFMessagePostProcessor processor = new RabbitService.LFMessagePostProcessor(message);
//Exchange:每个Virtual Host包含0或多个Exchange。Exchange负责把Message转发到Queue。每个Exchange可以有0或者多个Queue。每个Queue只能监听1个Exchange
//Routing Key:每个Message一般来说必须指定一个Routing Key,Exchange根据Message的Routing Key进行Binding,然后完成Message的转发
//发送缓存记录
this.sendCache.addMessage(excharge, routingKey, message);
this.template.convertAndSend(excharge, routingKey, message, processor, correlationData);
}
private class LFMessagePostProcessor implements MessagePostProcessor {
private MQMessage msg;
LFMessagePostProcessor(MQMessage message) {
this.msg = message;
}
public Message postProcessMessage(Message message) throws AmqpException {
if (this.msg != null && this.msg.getExpire() != null) {
message.getMessageProperties().setExpiration(String.valueOf(this.msg.getExpire()));
}
message.getMessageProperties().setHeader("message_id", this.msg.getId());
message.getMessageProperties().setTimestamp(new Date());
return message;
}
}
}
4.MQSendCache.java
@Component
public class MQSendCache implements InitService {
private static final Map sended = new ConcurrentHashMap();
private static final BlockingQueue queue = new DelayQueue();
public MQSendCache() {
}
public void addMessage(String excharge, String routeKey, MQMessage mqmessage) {
MQSendCache.SendMessage smessage = new MQSendCache.SendMessage(excharge, routeKey, mqmessage);
sended.put(mqmessage.getId(), smessage);
}
public void addFailure(String id) {
MQSendCache.SendMessage smessage = (MQSendCache.SendMessage)sended.remove(id);
smessage.setEndDate(System.currentTimeMillis());
queue.add(smessage);
}
public MQMessage get(String id) {
MQSendCache.SendMessage smessage = (MQSendCache.SendMessage)sended.get(id);
return smessage != null ? smessage.getMsg() : null;
}
public void removeMessage(String id) {
sended.remove(id);
}
public void init() {
RabbitService rabbitService = (RabbitService)SpringApplicationContext.getBean(RabbitService.class);
while(true) {
while(true) {
try {
MQSendCache.SendMessage smessage = (MQSendCache.SendMessage)queue.take();
if (smessage != null) {
MQMessage msg = smessage.getMsg();
//如果消息发送失败,则继续尝试,最多5次
if (msg.getSendTimes() <= 5) {
msg.setSendTimes(msg.getSendTimes() + 1);
rabbitService.send(smessage.getExcharge(), smessage.getRouteKey(), msg);
}
}
} catch (InterruptedException var4) {
var4.printStackTrace();
}
}
}
}
private class SendMessage implements Delayed {
private MQMessage msg;
private long endDate;
private String excharge;
private String routeKey;
SendMessage(String excharge, String routeKey, MQMessage msg) {
this.msg = msg;
this.endDate = System.currentTimeMillis();
this.excharge = excharge;
this.routeKey = routeKey;
}
public int compareTo(Delayed o) {
MQSendCache.SendMessage jia = (MQSendCache.SendMessage)o;
return this.endDate - jia.getEndDate() > 0L ? 1 : 0;
}
public long getDelay(TimeUnit unit) {
return this.endDate + 1000L - System.currentTimeMillis();
}
public MQMessage getMsg() {
return this.msg;
}
public long getEndDate() {
return this.endDate;
}
public void setEndDate(long endDate) {
this.endDate = endDate;
}
public String getExcharge() {
return this.excharge;
}
public String getRouteKey() {
return this.routeKey;
}
}
}
5.实现监听类
(1)消息队列具体的逻辑操作须在此Listener中实现;
@Service
public class IMMessageListener extends AbstractMQEventProcesser {
private static final Logger logger = LoggerFactory.getLogger(RequestIMServiceImpl.class);
@Value("${im.appKey}")
private String appKey;
@Value("${im.appSecret}")
private String appSecret;
@Override
public String event() {
//此处的event需要与发送消息时传递的event对应
return ChatConstant.MESSAGE_PARAMETER.ENENT;
}
@Override
public boolean precess(Object data) throws Exception {
Map map = (Map) data;
return getRequest((String) map.get(ChatConstant.MESSAGE_PARAMETER.URL), map.get(ChatConstant.MESSAGE_PARAMETER.VO));
}
public boolean getRequest(String url, Object vo) {
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpPost httpPost = new HttpPost(url);
String nonce = StringUtil.createRandom(false, 20);
String curTime = String.valueOf((new Date()).getTime() / 1000L);
String checkSum = ImCheckSum.getCheckSum(appSecret, nonce, curTime);//参考 计算CheckSum的java代码
try {
// 设置请求的header
httpPost.addHeader("AppKey", appKey);
httpPost.addHeader("Nonce", nonce);
httpPost.addHeader("CurTime", curTime);
httpPost.addHeader("CheckSum", checkSum);
httpPost.addHeader("Content-Type", "application/x-www-form-urlencoded;charset=utf-8");
Map map = (Map) vo;
List nvps = new ArrayList();
Iterator> entries = map.entrySet().iterator();
while (entries.hasNext()) {
Map.Entry entry = entries.next();
if (null == entry.getValue()) {
nvps.add(new BasicNameValuePair(entry.getKey(), null));
} else if (entry.getValue() instanceof String) {
nvps.add(new BasicNameValuePair(entry.getKey(), entry.getValue().toString()));
} else {
nvps.add(new BasicNameValuePair(entry.getKey(), JSONUtils.toJSONString(entry.getValue())));
}
}
httpPost.setEntity(new UrlEncodedFormEntity(nvps, "utf-8"));
logger.info("==========:" + url + new UrlEncodedFormEntity(nvps, "utf-8") + "==========:");
// 执行请求
HttpResponse response = httpClient.execute(httpPost);
//.entity所得到的流是不可重复读取的也就是说所得的到实体只能一次消耗完,
//不能多次读取,所以在执行html = EntityUtils.toString(entity)后,流就关闭了,就导致后面的读和写显示错误.
String responseString = EntityUtils.toString(response.getEntity());
logger.info("==========:" + responseString + "===========");
ResponseIM responeIM = JsonUtil.fromJson(responseString, ResponseIM.class);
if (!(responeIM.getCode() == ChatConstant.CODE.SUCCESS)) {
return false;
}
return true;
} catch (Exception e) {
return false;
}
}
}
Listener父类
public abstract class AbstractMQEventProcesser {
public AbstractMQEventProcesser() {
}
public abstract String event();
public abstract boolean precess(Object data) throws Exception;
}
6.队列配置
(1) config
@Configuration
@ConditionalOnProperty("spring.rabbitmq.host")
public class MQConfiguration {
@Value("${spring.application.name:}")
private String appname;
@Bean
public Queue queue() {
return new Queue(appname);
}
@Bean
public DirectExchange directExchange() {
return new DirectExchange(Excharge.direct.toString());
}
@Bean
public TopicExchange topicExchange() {
return new TopicExchange(Excharge.topic.toString());
}
@Bean
Binding bindingDirectExchange(Queue queue, DirectExchange directExchange) {
return BindingBuilder.bind(queue).to(directExchange).with(appname);
}
@Bean
Binding bindingTopicExchange(Queue queue, TopicExchange topicExchange) {
return BindingBuilder.bind(queue).to(topicExchange).with("#" + appname + "#");
}
@Bean
public RabbitService rabbitService() {
return new RabbitService();
}
@Bean
public MQSendCache mqsendCache() {
return new MQSendCache();
}
@Bean
@ConditionalOnProperty(value = "redis.config.host")
public ACKRedisCache ackRedisCache() {
return new ACKRedisCache();
}
@Bean
@ConditionalOnBean(name = "ackRedisCache")
ManualAckListener manualAckListenerWithCache(ACKRedisCache cache) {
ManualAckListener listener = new ManualAckListener();
listener.setCache(cache);
return listener;
}
@Bean
@ConditionalOnMissingBean(name = "ackRedisCache")
ManualAckListener manualAckListener() {
return new ManualAckListener();
}
@Autowired
public ConnectionFactory connectionFactory;
@Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory() {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setConcurrentConsumers(3);
factory.setMaxConcurrentConsumers(10);
return factory;
}
@Bean
public MessageConverter jsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
@Bean
LFConfirmCallback confirmCallback(){
return new LFConfirmCallback();
}
@Bean
LFReturnCallback returnCallback(){
return new LFReturnCallback();
}
@Bean
public RabbitTemplate rabbitTemplate() {
if(connectionFactory instanceof CachingConnectionFactory)
{
((CachingConnectionFactory) connectionFactory).setPublisherConfirms(true);
((CachingConnectionFactory) connectionFactory).setPublisherReturns(true);
}
RabbitTemplate template = new RabbitTemplate(connectionFactory);
template.setMessageConverter(jsonMessageConverter());
template.setConfirmCallback(confirmCallback());
template.setReturnCallback(returnCallback());
template.setMandatory(true);
return template;
}
}
(2)callback
@Component
public class LFConfirmCallback implements ConfirmCallback{
@Autowired
private MQSendCache sendCache;
@Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
if (!ack) {
// 发送失败
sendCache.addFailure(correlationData.getId());
} else {
// 发送成功
sendCache.removeMessage(correlationData.getId());
}
}
}
@Slf4j
@Component
public class LFReturnCallback implements ReturnCallback{
@Override
public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
log.error("消息发送异常,无路由信息: exchange:" + exchange + " routingKey:" + routingKey);
}
}
(3)listener
@Component
public abstract class AbstractRabbitManualAckListener implements ChannelAwareMessageListener {
private static final Logger log = LoggerFactory.getLogger(AbstractRabbitManualAckListener.class);
private ACKDefaultCache cache = new ACKDefaultCache();
@Value("${spring.application.name}")
private String queueName;
public AbstractRabbitManualAckListener() {
}
public abstract boolean received(Message message) throws Exception;
@RabbitListener(
queues = {"${spring.application.name}"}
)
public final void onMessage(Message message, Channel channel) throws Exception {
boolean success = false;
String messageId = StringUtil.value(message.getMessageProperties().getHeaders().get("message_id"));
success = this.messageConsumed(messageId);
if (!success) {
success = this.received(message);
}
if (success) {
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
this.setMessageCunsumed(messageId);
} else {
int times = this.messageFailure(messageId);
if (times > 5) {
channel.basicReject(message.getMessageProperties().getDeliveryTag(), false);
log.error("消息重发大于5次,将不再发送!");
return;
}
channel.basicReject(message.getMessageProperties().getDeliveryTag(), true);
}
}
public void setCache(ACKDefaultCache cache) {
this.cache = cache;
}
private boolean messageConsumed(String id) {
return this.cache.messageConsumed(id);
}
private void setMessageCunsumed(String id) {
this.cache.setMessageCunsumed(id);
}
private Integer messageFailure(String id) {
return this.cache.messageFailure(id);
}
}
@Component
public class ManualAckListener extends AbstractRabbitManualAckListener {
private static final Logger log = LoggerFactory.getLogger(ManualAckListener.class);
private Map map = null;
public ManualAckListener() {
}
public boolean received(Message message) {
try {
MQMessage msg = (MQMessage)JsonUtil.fromJson(new String(message.getBody()), MQMessage.class);
log.info("接收到消息: " + new String(message.getBody()));
String event = msg.getEvent();
AbstractMQEventProcesser process = (AbstractMQEventProcesser)this.getEventMap().get(event);
if (process == null) {
log.error("未找到事件:" + event + " 处理类");
return true;
} else {
return process.precess(msg.getData());
}
} catch (Exception var5) {
log.error("处理消息异常", var5);
return false;
}
}
private Map getEventMap() {
if (this.map == null) {
this.map = new HashMap();
Map processBeans = SpringApplicationContext.applicationContext().getBeansOfType(AbstractMQEventProcesser.class);
Collection processer = processBeans.values();
Iterator it = processer.iterator();
while(it.hasNext()) {
AbstractMQEventProcesser process = (AbstractMQEventProcesser)it.next();
this.map.put(process.event(), process);
}
}
return this.map;
}
}