哇，今天2019年最后一天上班，实在没心情撸代码，感觉隔了好久没写博客了，乘机写点吧。
回想2019实在是充实的一年，买房、迁户口、领结婚证、换工作啥都来了，说到换工作，实在想吐槽一下前东家，干了两年毛都没有捞到，技术成长几乎没有，加班到后半夜写业务，钱没赚到多，扣得一逼，入职时说期权，结果毛都没看一根，两年才涨500块，还他妈是入职时压下去的500，老板人前一套人后一套，空降的领导（外号做事情的人，后面都用这个外号替代）恶心一逼，甩锅抢功劳一流，欺下媚上，总算10月底逃离苦海了，虽然咱刚走就裁员三分之一，损失了几万补偿金也心甘情愿。
当初刚入职的时候想着创业公司50人，咱也算核心员工了，说是有期权，还蛮期待的，但是现在200-300人了，老员工走光了，项目被做事情的人架构得一塌糊涂。其实职场都差不多吧，但这么极品的人还真是第一次碰到，做事情的人客观来讲技术实力是有点，大概也就阿里P6-7的水平吧，但架子比P9都高的感觉，大概之前没当过领导，总有点小人得志的感觉。哎，算了吐槽这种人纯粹在浪费时间，这里建议各位同行还是能在大厂就尽量在大厂呆，福利待遇起码有保障，以后资历也有亮点，小厂真的就和赌博一样，可惜咱是赌输了，话说咱也从来没赌赢过。吐槽就这么多，下面上干活：
延迟队列是什么？
相信写过几年代码的应该都知道，不知道的自己去谷歌，我的理解：它其实是为了解决特点场景的一种技术方案。比如如下场景：
你下单购物，但订单20分钟未支付就要失效，你会怎么做？
可能刚入行的第一反应就是job定时扫啊，不错job定时扫是能解决问题，可是不够优雅，而且job的时间怎么设？如何保证准时性？（如果赶时间或者不要求准时性这些的话用xxl-job或者@Scheduled(cron = "")注解通过job定时扫也可以实现就不说了。）
这时延迟队列的特点就可以完美应用于该场景，当下单时往延迟队列放一条，20分钟后执行订单失效操作。据我所知道延迟队列有以下几种实现方式：
1、基于MQ实现：
参考下面博客：
https://www.cnblogs.com/hzmark/p/mq-delay-msg.html
https://blog.csdn.net/qq_21033663/article/details/101222502
阿里云的RocketMQ ，具体可以参考官方文档：https://help.aliyun.com/document_detail/43349.html
大概思路是下图：
MQ实现延迟队列.png
2、基于Redis实现：
这个就多了，首先目前最火的Redis框架Redisson就有这种解决方案，事实上这个框架基本上把Redis都玩出花了，分布式锁、分布式对象、分布式集合、分布式服务等都包含了。这里不多介绍，自己去GitHub上看：https://github.com/redisson/redisson
下面主要介绍新公司的大佬基于lua脚本和Guava-EventBus的简单实现，来，一起膜拜一波：
目录结构.png
DelayQProducer：
package com.company.project.service.util.delayqueue.producer;
import com.company.project.service.util.delayqueue.DelayJob;
import java.time.Duration;
public interface DelayQProducer {
/**
* 提交任务，若存在则更新
*/
void submit(String queue, DelayJob job, Duration delayed);
/**
* 更新任务，不存在返回false
*/
boolean update(String queue, DelayJob job, Duration delayed);
/**
* 取消任务
*/
void cancel(String queue, DelayJob job);
/**
* 查询任务队列长度
*/
default int getQueueSize(String queue) {
return 0;
}
String getNamespace();
}
DelayQProducer：
package com.company.project.service.util.delayqueue.producer;
import com.alibaba.fastjson.JSON;
import com.company.project.service.util.PooledRedisClient;
import com.company.project.service.util.delayqueue.DelayJob;
import lombok.Getter;
import lombok.Setter;
import java.time.Duration;
import java.time.Instant;
import static com.company.project.service.util.delayqueue.util.DQUtil.buildKey;
public class DelayQProducerImpl implements DelayQProducer {
@Setter
@Getter
private String namespace;
@Setter
private PooledRedisClient pooledRedisClient;
@Override
public void submit(String queue, DelayJob job, Duration delayed) {
double score = Instant.now().toEpochMilli() + delayed.toMillis();
pooledRedisClient.runWithRetry(jedis -> jedis.zadd(buildKey(namespace, queue), score, JSON.toJSONString(job)));
}
@Override
public boolean update(String queue, DelayJob job, Duration delayed) {
String jobString = JSON.toJSONString(job);
Double oldScore = pooledRedisClient.executeWithRetry(jedis -> jedis.zscore(buildKey(namespace, queue), jobString));
if (oldScore == null) {
return false;
} else {
double newScore = Instant.now().toEpochMilli() + delayed.toMillis();
pooledRedisClient.runWithRetry(jedis -> jedis.zadd(buildKey(namespace, queue), newScore, JSON.toJSONString(job)));
return true;
}
}
@Override
public void cancel(String queue, DelayJob job) {
pooledRedisClient.runWithRetry(jedis -> jedis.zrem(buildKey(namespace, queue), JSON.toJSONString(job)));
}
@Override
public int getQueueSize(String queue) {
return pooledRedisClient.executeWithRetry(jedis -> jedis.zcard(buildKey(namespace, queue))).intValue();
}
}
依赖PooledRedisClient：
package com.company.project.service.util;
import lombok.extern.slf4j.Slf4j;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import java.util.function.Consumer;
import java.util.function.Function;
@Slf4j
public class PooledRedisClient {
private static final int DEFAULT_RETRY_TIME = 3;
private static final int MIN_RETRY_TIME = 0;
private static final int MAX_RETRY_TIME = 5;
private static final long REDIS_EXECUTE_WARN_TIME = 50;
private final JedisPool pool;
public PooledRedisClient(JedisPool pool) {
this.pool = pool;
}
public void run(Consumer command) {
execute(jedis -> {
command.accept(jedis);
return null;
});
}
public void runWithRetry(Consumer command) {
runWithRetry(command, DEFAULT_RETRY_TIME);
}
public void runWithRetry(Consumer command, int retries) {
executeWithRetry(jedis -> {
command.accept(jedis);
return null;
}, retries);
}
public T execute(Function command) {
long startTime = System.currentTimeMillis();
Jedis jedis = null;
try {
jedis = this.pool.getResource();
return command.apply(jedis);
} finally {
if (jedis != null) {
jedis.close();
}
long endTime = System.currentTimeMillis();
long useTime = endTime - startTime;
if (useTime > REDIS_EXECUTE_WARN_TIME) {
log.warn("pooled redis client execute time warn. useTime:[{}],command:[{}]", useTime, command);
}
}
}
public T executeWithRetry(Function command) {
return executeWithRetry(command, DEFAULT_RETRY_TIME);
}
public T executeWithRetry(Function command, int retryTime) {
if (retryTime <= MIN_RETRY_TIME || retryTime > MAX_RETRY_TIME) {
retryTime = DEFAULT_RETRY_TIME;
}
int failTime = 0;
T result;
while (true) {
long startTime = System.currentTimeMillis();
try (Jedis jedis = this.pool.getResource()) {
log.debug("pooled redis client execute start.command:[{}]", command);
result = command.apply(jedis);
break;
} catch (RuntimeException e) {
failTime++;
if (failTime >= retryTime) {
throw e;
}
} finally {
long useTime = System.currentTimeMillis() - startTime;
if (useTime > REDIS_EXECUTE_WARN_TIME) {
log.warn("pooled redis client execute time warn. useTime:[{}],command:[{}]", useTime, command);
}
}
}
return result;
}
public JedisPool getPool() {
return this.pool;
}
}
BoundedDelayQProducer：
package com.company.project.service.util.delayqueue.producer;
import com.company.project.service.util.delayqueue.DelayJob;
import lombok.Setter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.Duration;
public class BoundedDelayQProducer implements DelayQProducer {
private final Logger logger = LoggerFactory.getLogger(getClass());
@Setter
private DelayQProducer delegate;
/**
* warning: 按每个job 512字节计算，200,000个job将占用redis约100MB内存
* 后续需要考虑对queue分片
*/
@Setter
private int maxQueueSize = 200000;
@Setter
private int warningSize = 100000;
@Override
public void submit(String queue, DelayJob job, Duration delayed) {
int queueSize = delegate.getQueueSize(queue);
if (queueSize > maxQueueSize) {
logger.error("task queue:{} exceed limit:{}, task discard", queue, maxQueueSize);
} else if (queueSize > warningSize) {
logger.warn("task queue:{} size warning, current size:{}", queue, queueSize);
delegate.submit(queue, job, delayed);
} else {
logger.debug("task queue:{}, size:{}", queue, queueSize);
delegate.submit(queue, job, delayed);
}
}
@Override
public boolean update(String queue, DelayJob job, Duration delayed) {
return delegate.update(queue, job, delayed);
}
@Override
public void cancel(String queue, DelayJob job) {
delegate.cancel(queue, job);
}
@Override
public String getNamespace() {
return delegate.getNamespace();
}
}
DelayQConsumer:
package com.company.project.service.util.delayqueue.consumer;
public interface DelayQConsumer {
void subscribe(String queue);
}
DelayQConsumerImpl:
package com.company.project.service.util.delayqueue.consumer;
import com.alibaba.fastjson.JSON;
import com.company.project.service.util.PooledRedisClient;
import com.company.project.service.util.delayqueue.DelayJob;
import com.company.project.service.util.delayqueue.DelayJobProcessor;
import com.company.project.service.util.delayqueue.util.DQUtil;
import com.google.common.eventbus.AsyncEventBus;
import lombok.Setter;
import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.concurrent.BasicThreadFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.DisposableBean;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import java.time.Instant;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
@SuppressWarnings("UnstableApiUsage")
public class DelayQConsumerImpl implements DelayQConsumer, ApplicationContextAware, DisposableBean, InitializingBean {
private final Logger logger = LoggerFactory.getLogger(getClass());
private static final String POP_SCRIPT = "local rlt = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', ARGV[1], 'LIMIT', '0', ARGV[2])"
+ " if next(rlt) ~= nil then redis.call('ZREM', KEYS[1], unpack(rlt)) end"
+ " return rlt";
private static final long SLEEP_FOR_EMPTY_QUEUE = 1000L;
private static final long SLEEP_FOR_ERROR = 5000L;
private static final int MAX_IN_FLIGHT_JOB = 2000;
private static final int EVENT_BUS_THREAD_POOL_SIZE = 8;
@Setter
private String namespace;
@Setter
private String initWithQueue;
@Setter
private String batchSize = "10";
@Setter
private PooledRedisClient pooledRedisClient;
private volatile State state = State.NEW;
private AsyncEventBus asyncEventBus;
private ExecutorService eventBusPool;
@Override
public void subscribe(String queue) {
if (state == State.TERMINATED) {
return;
}
logger.info("subscribe " + queue + " on namespace " + namespace);
state = State.RUNNING;
new Thread(new Consumer(queue)).start();
}
@Override
public void afterPropertiesSet() throws Exception {
if (StringUtils.isNotBlank(initWithQueue)) {
this.subscribe(initWithQueue);
}
}
@Override
public void destroy() throws Exception {
state = State.TERMINATED;
eventBusPool.shutdown();
if (!eventBusPool.awaitTermination(30, TimeUnit.SECONDS)) {
logger.warn("force shutdown delay queue consumer eventBusPool");
eventBusPool.shutdownNow();
}
}
@SuppressWarnings("rawtypes")
class Consumer implements Runnable {
private String queue;
private Map classCache = new HashMap<>();
public Consumer(String queue) {
this.queue = queue;
}
@Override
public void run() {
Thread.currentThread().setName("delay-queue-consumer-thread-" + namespace + "-" + queue);
String key = DQUtil.buildKey(namespace, queue);
while (state == State.RUNNING) {
try {
String now = String.valueOf(Instant.now().toEpochMilli());
List jobs = null;
try {
jobs = (List) pooledRedisClient.executeWithRetry(jedis -> jedis.eval(POP_SCRIPT, 1, key, now, batchSize));
} catch (Exception e) {
logger.error("redis error, namespace:{}, queue:{}", namespace, queue, e);
}
if (CollectionUtils.isEmpty(jobs)) {
Thread.sleep(SLEEP_FOR_EMPTY_QUEUE);
continue;
}
for (Object o : jobs) {
DelayJob job = JSON.parseObject(o.toString(), DelayJob.class);
Object jobEntity = JSON.toJavaObject((JSON) job.getEntity(), getClass(job.getClassName()));
logger.info("receive delayed job:{}", jobEntity);
asyncEventBus.post(jobEntity);
}
} catch (Throwable e) {
logger.error("unexpected consume error, namespace:{}, queue:{}", namespace, queue, e);
try {
Thread.sleep(SLEEP_FOR_ERROR);
} catch (InterruptedException ignore) {
}
}
}
logger.info("consumer terminated, namespace:{}, queue:{}", namespace, queue);
}
private Class getClass(String className) throws ClassNotFoundException {
Class clazz = classCache.get(className);
if (clazz == null) {
clazz = Class.forName(className);
classCache.put(className, clazz);
}
return clazz;
}
}
@Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
// AsyncEventBus内部使用无界队列存储事件，无法在post阶段限流，只能在消费时丢弃，目前没有溢出风险，后续考虑更换为mbassador
eventBusPool = new ThreadPoolExecutor(EVENT_BUS_THREAD_POOL_SIZE, EVENT_BUS_THREAD_POOL_SIZE, 0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<>(MAX_IN_FLIGHT_JOB),
new BasicThreadFactory.Builder().namingPattern("event-bus-delay-queue-consumer-%d").build(),
(r, executor) -> logger.error("delay task discarded due to overflow"));
asyncEventBus = new AsyncEventBus(eventBusPool, (exception, context) -> logger.error("async event process failed with subscriber: {} method: {} event: {}",
context.getSubscriber().getClass().getSimpleName(), context.getSubscriberMethod().getName(), context.getEvent(), exception));
applicationContext.getBeansWithAnnotation(DelayJobProcessor.class).values().forEach(bean -> {
logger.info("register delay queue subscriber: {}", bean.getClass().getSimpleName());
asyncEventBus.register(bean);
});
}
public enum State {
NEW, RUNNING, TERMINATED
}
}
DQUtil:
package com.company.project.service.util.delayqueue.util;
import com.google.common.base.Preconditions;
import org.apache.commons.lang3.StringUtils;
public class DQUtil {
private static final String KEY_PREFIX = "card_im_delay_queue";
private static final String DELIMITER = ":";
public static String buildKey(String namespace, String key) {
Preconditions.checkArgument(StringUtils.isNotBlank(namespace));
Preconditions.checkArgument(StringUtils.isNotBlank(key));
return KEY_PREFIX + DELIMITER + namespace + DELIMITER + key;
}
public static String buildKey(String key) {
return KEY_PREFIX + DELIMITER + key;
}
}
DelayJob:
package com.company.project.service.util.delayqueue;
import lombok.Data;
import java.io.Serializable;
import java.util.HashMap;
import java.util.Map;
@Data
public class DelayJob implements Serializable {
private static final long serialVersionUID = -2392357322522404381L;
private String className;
private T entity;
private Map context;
public DelayJob() {
}
public DelayJob(T entity) {
this.entity = entity;
this.className = entity.getClass().getName();
this.context = new HashMap<>();
}
public DelayJob(T entity, Map context) {
this.entity = entity;
this.className = entity.getClass().getName();
this.context = context;
}
}
DelayJobProcessor:
package com.company.project.service.util.delayqueue;
import org.springframework.stereotype.Component;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
/**
* for auto register EventBus subscriber
*/
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Component
public @interface DelayJobProcessor {
}
DelayQConfig:
package com.company.project.service.util.delayqueue;
import com.company.project.service.util.PooledRedisClient;
import com.company.project.service.util.delayqueue.consumer.DelayQConsumer;
import com.company.project.service.util.delayqueue.consumer.DelayQConsumerImpl;
import com.company.project.service.util.delayqueue.producer.BoundedDelayQProducer;
import com.company.project.service.util.delayqueue.producer.DelayQProducer;
import com.company.project.service.util.delayqueue.producer.DelayQProducerImpl;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import javax.annotation.Resource;
@Configuration
public class DelayQConfig {
/**
* 交互事件超时结束任务延迟队列
*/
public static final String DELAY_QUEUE_INTERACT_EVENT_AUTO_CLOSE = "interact_event_auto_close";
@Resource
private PooledRedisClient pooledRedisClient;
@Value(value = "${business-card.delayqueue.namespace:business-card}")
private String namespace;
@Bean
public DelayQProducer delayQProducer() {
DelayQProducerImpl delayQProducer = new DelayQProducerImpl();
delayQProducer.setNamespace(namespace);
delayQProducer.setPooledRedisClient(pooledRedisClient);
BoundedDelayQProducer boundedProducer = new BoundedDelayQProducer();
boundedProducer.setDelegate(delayQProducer);
return boundedProducer;
}
@Bean
public DelayQConsumer delayQConsumer() {
DelayQConsumerImpl consumer = new DelayQConsumerImpl();
consumer.setInitWithQueue(DELAY_QUEUE_INTERACT_EVENT_AUTO_CLOSE);
consumer.setNamespace(namespace);
consumer.setPooledRedisClient(pooledRedisClient);
return consumer;
}
}
好了，怎么用呢？
如下：
提交事件（生产）.png
到期执行（消费）.png
哎,原本想着趁过年放假在家好好进修看书看视频，然而果然还是没那自制力，这次新型冠状病毒疫情让我从1月24日到今天2月14日快在家呆一个月，人都快发霉了，接下来要立flag:
新的一年要看完这个链接里全部的视频：https://www.bilibili.com/video/av79788004
书的话要把现在有的基本都看一遍至少：
《第一本docker书》
《高性能MYSQL》
《领域驱动设计》
《Java虚拟机精讲》
《设计模式》
《Spring源码精析》
《Spring Cloud与微服务构建》
学习了解ServiceMesh与K8S,我相信这将是这两年的必备技术。
然后愿接下来的十年一次的经济危机温柔点！