Redis 5 新特性中,Streams 数据结构的引入,可以说它是在本次迭代中最大特性。它使本次 5.x 版本迭代中,Redis 作为消息队列使用时,得到更完善,更强大的原生支持,其中尤为明显的是持久化消息队列。同时,stream 借鉴了 kafka 的消费组模型概念和设计,使消费消息处理上更加高效快速。
Redis中有三种消息队列模式:
名称 |
简要说明 |
List |
不支持消息确认机制(Ack),不支持消息回朔 |
pubSub |
不支持消息确认机制(Ack),不支持消息回朔,不支持消息持久化 |
stream |
支持消息确认机制(Ack),支持消息回朔,支持消息持久化,支持消息阻塞 |
stream消息队列相关命令:
XADD - 添加消息到末尾
XTRIM - 对流进行修剪,限制长度
XDEL - 删除消息
XLEN - 获取流包含的元素数量,即消息长度
XRANGE - 获取消息列表,会自动过滤已经删除的消息
XREVRANGE - 反向获取消息列表,ID 从大到小
XREAD - 以阻塞或非阻塞方式获取消息列表
消费者组相关命令:
XGROUP CREATE - 创建消费者组
XREADGROUP GROUP - 读取消费者组中的消息
XACK - 将消息标记为"已处理"
XGROUP SETID - 为消费者组设置新的最后递送消息ID
XGROUP DELCONSUMER - 删除消费者
XGROUP DESTROY - 删除消费者组
XPENDING - 显示待处理消息的相关信息
XCLAIM - 转移消息的归属权
XINFO - 查看流和消费者组的相关信息;
XINFO GROUPS - 打印消费者组的信息;
XINFO STREAM - 打印流信息
有几个常见问题:
(1)消息拉取成功但是消费失败,如何做到不丢失数据
(2)如果数据未消费完,redis宕机了,如何做到数据不丢失?---持久化
stream作为redis数据类型的一种,它的每个写操作也都会被AOF记录下来, 写入的结果也会被RDB记录下.AOF
记录了redis写操作的操作历史RDB
则是根据一定规则对redis内存中的数据做快照
如果redis宕机重启后,如果配置好持久化策略,也能够恢复回来
但是
专业的消息中间件,比如Apach Kafka有集群,副本和leader的概念, 每个节点(broker)数据改变都会往其他节点上更新副本, 这样的话,只要保证集群中数据最完整,响应速度最快的那个节点作为主节点(leader),就最大可能性保证数据不完整了
(3)消息积压了怎么办?
消息中间件就像一个水池, 生产者是进入口,消费者是出水口.如果出水的速度比进水慢,那么就会造成消息积压.
解决积压的两个常规思路:
a. 限制生产者生产消息的速度
比如如果是web项目,我们可以通过限流来限制客户访问的数量, 超出数量的客户就提示他网站正忙,稍后重试.
b. 增加消费者消费速度
有一些场景是无法限制生产者生产速度的, 比如接受工厂机器传感器监控生产而定期传入的数据,这些数据是用来控制产品质量的,必须按照一定的并发量生产消息.增加多消费者的方式
stream怎么解决处理:
因为redis的数据都放在内存中, 消息积压可能会导致内存溢出. 所以stream有一个属性就是队列最大长度(MAXLEN), 如果消息积压超过了最大长度,最旧的消息会被截断(XTRIM)丢掉.
具体操作是:
#创建时指定最大长度
XADD stream10 MAXLEN 1000 * field1 value1
"1691035235169-0"
java springboot项目如何操作redis stream呢
数据准备:
建立stream和对应的消费者组
打开redis-cli.exe客户端
#新增流
XADD dcir * data 1
XADD formation * data 1
XADD preCharge * data 1
XADD division * data 1
#新增消费者组,从末尾开始消费
XGROUP CREATE dcir dcir-group-1 $
XGROUP CREATE formation formation-group-1 $
XGROUP CREATE preCharge preCharge-group-1 $
XGROUP CREATE division division-group-1 $
org.springframework.boot
spring-boot-starter-parent
2.5.4
1.8
org.springframework.boot
spring-boot-starter-data-redis
org.apache.commons
commons-pool2
消息生产者:
spring:
redis:
host: 10.168.204.80
database: 0
port: 6379
password: 123456
timeout: 1000
lettuce:
pool:
max-active: 8
max-wait: -1
max-idle: 8
min-idle: 0
server:
port: 8087
redisstream:
stream: dcir
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
@Data
@Component
@ConfigurationProperties(prefix = "redisstream")
public class RedisStreamConfig {
private String stream;
}
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.connection.stream.StreamRecords;
import org.springframework.data.redis.connection.stream.StringRecord;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Service;
import java.util.Collections;
@Service
@Slf4j
public class RedisPushService {
@Autowired
private StringRedisTemplate stringRedisTemplate;
@Autowired
private RedisStreamConfig redisStreamConfig;
public void push(String msg){
// 创建消息记录, 以及指定stream
StringRecord stringRecord = StreamRecords.string(Collections.singletonMap("data", msg)).withStreamKey(redisStreamConfig.getStream());
// 将消息添加至消息队列中
this.stringRedisTemplate.opsForStream().add(stringRecord);
log.info("{}已发送消息:{}",redisStreamConfig.getStream(),msg);
}
}
消费者:
spring:
redis:
database: 0
host: 10.168.204.80
port: 6379
password: 123456
timeout: 5000
jedis:
pool:
max-idle: 10
max-active: 50
max-wait: 1000
min-idle: 1
redisstream:
dcirgroup: dcir-group-1
dcirconsumer: dcir-consumer-1
formationgroup: formation-group-1
formationconsumer: formation-consumer-1
divisiongroup: division-group-1
divisionconsumer: division-consumer-1
prechargegroup: precharge-group-1
prechargeconsumer: precharge-consumer-1
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
@Data
@Component
@ConfigurationProperties(prefix = "redisstream")
public class RedisStreamConfig {
static final String DCIR = "dcir";
static final String PRECHARGE = "preCharge";
static final String FORMATION = "formation";
static final String DIVISION = "division";
private String stream;
private String group;
private String consumer;
private String dcirgroup;
private String formationgroup;
private String divisiongroup;
private String prechargegroup;
private String dcirconsumer;
private String formationconsumer;
private String divisionconsumer;
private String prechargeconsumer;
}
import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.PropertyAccessor;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.connection.stream.Consumer;
import org.springframework.data.redis.connection.stream.ObjectRecord;
import org.springframework.data.redis.connection.stream.ReadOffset;
import org.springframework.data.redis.connection.stream.StreamOffset;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;
import org.springframework.data.redis.stream.StreamListener;
import org.springframework.data.redis.stream.StreamMessageListenerContainer;
import java.time.Duration;
import java.util.concurrent.ExecutorService;
@Configuration
@Slf4j
public class RedisStreamConsumerConfig {
@Autowired
ExecutorService executorService;
@Autowired
RedisStreamConfig redisStreamConfig;
/**
* 主要做的是将OrderStreamListener监听绑定消费者,用于接收消息
*
* @param connectionFactory
* @param streamListener
* @return
*/
@Bean
public StreamMessageListenerContainer> dcirConsumerListener(
RedisConnectionFactory connectionFactory,
DcirStreamListener streamListener) {
StreamMessageListenerContainer> container =
streamContainer(redisStreamConfig.DCIR, connectionFactory, streamListener);
container.start();
return container;
}
@Bean
public StreamMessageListenerContainer> divisionConsumerListener(
RedisConnectionFactory connectionFactory,
DivisionStreamListener streamListener) {
StreamMessageListenerContainer> container =
streamContainer(redisStreamConfig.DIVISION, connectionFactory, streamListener);
container.start();
return container;
}
@Bean
public StreamMessageListenerContainer> formationConsumerListener(
RedisConnectionFactory connectionFactory,
FormationStreamListener streamListener) {
StreamMessageListenerContainer> container =
streamContainer(redisStreamConfig.FORMATION, connectionFactory, streamListener);
container.start();
return container;
}
@Bean
public StreamMessageListenerContainer> preChargeConsumerListener(
RedisConnectionFactory connectionFactory,
PrechargeStreamListener streamListener) {
StreamMessageListenerContainer> container =
streamContainer(redisStreamConfig.PRECHARGE, connectionFactory, streamListener);
container.start();
return container;
}
/**
* @param mystream 从哪个流接收数据
* @param connectionFactory
* @param streamListener 绑定的监听类
* @return
*/
private StreamMessageListenerContainer> streamContainer(String mystream, RedisConnectionFactory connectionFactory, StreamListener> streamListener) {
StreamMessageListenerContainer.StreamMessageListenerContainerOptions> options =
StreamMessageListenerContainer.StreamMessageListenerContainerOptions
.builder()
.pollTimeout(Duration.ofSeconds(5)) // 拉取消息超时时间
.batchSize(10) // 批量抓取消息
.targetType(String.class) // 传递的数据类型
.executor(executorService)
.build();
StreamMessageListenerContainer> container = StreamMessageListenerContainer
.create(connectionFactory, options);
//指定消费最新的消息
StreamOffset offset = StreamOffset.create(mystream, ReadOffset.lastConsumed());
//创建消费者
StreamMessageListenerContainer.StreamReadRequest streamReadRequest = null;
try {
streamReadRequest = buildStreamReadRequest(offset, streamListener);
} catch (Exception e) {
log.error(e.getMessage());
}
//指定消费者对象
container.register(streamReadRequest, streamListener);
return container;
}
private StreamMessageListenerContainer.StreamReadRequest buildStreamReadRequest(StreamOffset offset, StreamListener> streamListener) throws Exception {
Consumer consumer = null;
if(streamListener instanceof DcirStreamListener){
consumer = Consumer.from(redisStreamConfig.getDcirgroup(), redisStreamConfig.getDcirconsumer());
}else if(streamListener instanceof DivisionStreamListener){
consumer = Consumer.from(redisStreamConfig.getDivisiongroup(), redisStreamConfig.getDivisionconsumer());
}else if(streamListener instanceof FormationStreamListener){
consumer = Consumer.from(redisStreamConfig.getFormationgroup(), redisStreamConfig.getFormationconsumer());
}else if(streamListener instanceof PrechargeStreamListener){
consumer = Consumer.from(redisStreamConfig.getPrechargegroup(), redisStreamConfig.getPrechargeconsumer());
}else{
throw new Exception("无法识别的stream key");
}
StreamMessageListenerContainer.StreamReadRequest streamReadRequest = StreamMessageListenerContainer.StreamReadRequest.builder(offset)
.errorHandler((error) -> {
error.printStackTrace();
log.error(error.getMessage());
})
.cancelOnError(e -> false)
.consumer(consumer)
//关闭自动ack确认
.autoAcknowledge(false)
.build();
return streamReadRequest;
}
@Bean
public RedisTemplate redisTemplate(RedisConnectionFactory factory) {
RedisTemplate template = new RedisTemplate();
template.setConnectionFactory(factory);
Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);
ObjectMapper om = new ObjectMapper();
om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
om.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);
jackson2JsonRedisSerializer.setObjectMapper(om);
StringRedisSerializer stringRedisSerializer = new StringRedisSerializer();
// key采用String的序列化方式
template.setKeySerializer(stringRedisSerializer);
// hash的key也采用String的序列化方式
template.setHashKeySerializer(stringRedisSerializer);
// value序列化方式采用jackson
template.setValueSerializer(jackson2JsonRedisSerializer);
// hash的value序列化方式采用jackson
template.setHashValueSerializer(jackson2JsonRedisSerializer);
template.afterPropertiesSet();
return template;
}
}
import com.alibaba.fastjson.JSONObject;
import com.qds.k2h.domain.*;
import com.qds.k2h.service.DcirDataService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.connection.stream.ObjectRecord;
import org.springframework.data.redis.connection.stream.RecordId;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.stream.StreamListener;
import org.springframework.stereotype.Component;
@Component
@Slf4j
public class DcirStreamListener implements StreamListener> {
@Autowired
StringRedisTemplate stringRedisTemplate;
@Autowired
RedisStreamConfig redisStreamConfig;
@Autowired
DcirDataService dcirDataService;
@Override
protected void finalize() throws Throwable {
super.finalize();
}
@Override
public void onMessage(ObjectRecord message) {
try{
// 消息ID
RecordId messageId = message.getId();
// 消息的key和value
String string = message.getValue();
log.info("dcir获取到数据。messageId={}, stream={}, body={}", messageId, message.getStream(), string);
DcirData data = JSONObject.parseObject(string, DcirData.class);
//业务逻辑
handle(data);
// 通过RedisTemplate手动确认消息
this.stringRedisTemplate.opsForStream().acknowledge(redisStreamConfig.getDcirgroup(), message);
}catch (Exception e){
// 处理异常
e.printStackTrace();
}
}
}
至此已完成相关的redis实现mq的功能。不过这种用法的话,还是适合简单且数据量小的数据传输之间,如果是项目之间的数据传输还是建议用主流的消息队列来进行实现