nested exception is io.lettuce.core.RedisCommandInterruptedException: Command interrupted

使用的工具

为springboot提供的工具

<dependency>
    <groupId>org.springframework.bootgroupId>
    <artifactId>spring-boot-starter-data-redisartifactId>
dependency>

问题

生产环境一直持续出现Redis command interrupted; nested exception is io.lettuce.core.RedisCommandInterruptedException: Command interrupted错误

详细的错误信息

org.springframework.data.redis.RedisSystemException: Redis command interrupted; nested exception is io.lettuce.core.RedisCommandInterruptedException: Command interrupted
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:62) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:275) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.convertLettuceAccessException(LettuceStringCommands.java:799) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.set(LettuceStringCommands.java:148) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.connection.DefaultedRedisConnection.set(DefaultedRedisConnection.java:287) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.connection.DefaultStringRedisConnection.set(DefaultStringRedisConnection.java:974) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.core.DefaultValueOperations$3.inRedis(DefaultValueOperations.java:240) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:60) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:228) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:188) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:96) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at org.springframework.data.redis.core.DefaultValueOperations.set(DefaultValueOperations.java:236) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    at com.test.service.TestService.lambda$test$19(TestService.java:532) ~[classes/:?]
    at java.lang.Thread.run(Thread.java:750) [?:1.8.0_361]
Caused by: io.lettuce.core.RedisCommandInterruptedException: Command interrupted
    at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:138) ~[lettuce-core-5.3.7.RELEASE.jar:5.3.7.RELEASE]
    at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:75) ~[lettuce-core-5.3.7.RELEASE.jar:5.3.7.RELEASE]
    at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:79) ~[lettuce-core-5.3.7.RELEASE.jar:5.3.7.RELEASE]
    at com.sun.proxy.$Proxy282.set(Unknown Source) ~[?:?]
    at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.set(LettuceStringCommands.java:146) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    ... 10 more
Caused by: java.lang.InterruptedException
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347) ~[?:1.8.0_361]
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_361]
    at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:121) ~[lettuce-core-5.3.7.RELEASE.jar:5.3.7.RELEASE]
    at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:75) ~[lettuce-core-5.3.7.RELEASE.jar:5.3.7.RELEASE]
    at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:79) ~[lettuce-core-5.3.7.RELEASE.jar:5.3.7.RELEASE]
    at com.sun.proxy.$Proxy282.set(Unknown Source) ~[?:?]
    at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.set(LettuceStringCommands.java:146) ~[spring-data-redis-2.3.9.RELEASE.jar:2.3.9.RELEASE]
    ... 10 more

注意这行错误Caused by: java.lang.InterruptedException at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347) ~[?:1.8.0_361]

CompletableFuture是jdk8提供的并发编程工具类，用于解决多个future阻塞问题

CompletableFuture#reportGet

private static <T> T reportGet(Object r)
    throws InterruptedException, ExecutionException {
    if (r == null) // by convention below, null means interrupted
        throw new InterruptedException();
    if (r instanceof AltResult) {
        Throwable x, cause;
        if ((x = ((AltResult)r).ex) == null)
            return null;
        if (x instanceof CancellationException)
            throw (CancellationException)x;
        if ((x instanceof CompletionException) &&
            (cause = x.getCause()) != null)
            x = cause;
        throw new ExecutionException(x);
    }
    @SuppressWarnings("unchecked") T t = (T) r;
    return t;
}

当线程发生interrupt中断后，r为null抛出中断异常。

直到调用到lettuce驱动后进行抛出RedisCommandInterruptedException异常

LettuceFutures#awaitOrCancel

public static <T> T awaitOrCancel(RedisFuture<T> cmd, long timeout, TimeUnit unit) {

    try {
        if (timeout > 0 && !cmd.await(timeout, unit)) {
            cmd.cancel(true);
            throw ExceptionFactory.createTimeoutException(Duration.ofNanos(unit.toNanos(timeout)));
        }
        return cmd.get();
       //省略部分无关代码... 
    }  catch (InterruptedException e) {

        Thread.currentThread().interrupt();
        throw new RedisCommandInterruptedException(e);
    } catch (Exception e) {
        throw ExceptionFactory.createExecutionException(null, e);
    }
}

到这里分析出原因是由于操作RedisTemplate的线程出现了中断，从而最终导致抛出了RedisCommandInterruptedException异常

那么是什么原因导致线程会被中断呢，下面来完整的分析业务代码

public void test(){
    new Thread(() -> {
        while (true) {
            try {
                String value = (String)redisTemplate.opsForValue().get("test1");           
                try {
                    //这里调用别的service方法对数据库进行操作
                	String result = otherService.op(value);
                }catch (Exception e) {
                    logger.info(e.toString());
                    Thread.currentThread().interrupt();
                }
                redisTemplate.delete("test1");   
            }catch (Exception e) {
                logger.info("test error",e);
            }
        }
    }).start();
}

String result = otherService.op(id);这行是对数据库进行操作，分析到可能是此处的原因，接下来去生产的ELK上查询此处的错误信息发现org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: interrupt存在这样的报错信息，大意是不能获取到jdbc的连接了。

到这里就知道完整的原因了：

1. 由于操作数据库时太频繁导致连接池连接不足
2. 当执行到String result = otherService.op(id);时就出抛出获取不到连接的异常
3. 然后此异常信息被catch捕获住，执行Thread.currentThread().interrupt()线程中断
4. 当再次循环到String value = (String)redisTemplate.opsForValue().get("test1");由于线程被中断，从而抛出RedisCommandInterruptedException异常

解决办法

1. 调大数据库连接池参数
2. 将catch Exception下的Thread.currentThread().interrupt()这行去掉
3. 当redisTemplate.opsForValue().get("test1");获取不到数据进行休眠一段时间然后再次循环