项目使用Azure提供的redis缓存服务,azure暴露一个redis连接地址,但是Azure内部实现是主备结构。由于azure redis缓存所在机器操作系统升级等情况会发生主备切换,造成redis客户端建立的连接失效,操作redis时会抛出两种类型的异常:RedisConnectionFailureException|JedisConnectionException
, 对服务造成影响。
客户端使用的连接池是:
org.springframework.data.redis.connection.jedis.JedisConnectionFactory
主要配置如下:
<property name="hostName" value="${REDIS_IP}"/>
<property name="port" value="${REDIS_PORT}"/>
<property name="database" value="${REDIS_DB}"/>
<property name="password" value="${REDIS_PASSWORD}"/>
<property name="poolConfig">
<bean class="redis.clients.jedis.JedisPoolConfig">
<property name="maxIdle" value="999"/>
<property name="maxTotal" value="9999"/>
<property name="minIdle" value="20"/>
bean>
property>
由于要保证redis的性能,连接池配置没有加testOnBorrow
(testOnBorrow每次获取redis连接时都会验证连接是否可用,保证每次拿到的redis连接都是有效的。).
<bean class="redis.clients.jedis.JedisPoolConfig">
<property name="maxIdle" value="999"/>
<property name="maxTotal" value="9999"/>
<property name="minIdle" value="20"/>
bean>
实际操作redis的类是spring提供的redistemplate类:
org.springframework.data.redis.core.RedisTemplate
这个类依赖redis连接池,所有操作redis的方法,最终通过execute:
public <T> T execute(RedisCallback<T> action, boolean exposeConnection, boolean pipeline) {
Assert.isTrue(this.initialized, "template not initialized; call afterPropertiesSet() before using it");
Assert.notNull(action, "Callback object must not be null");
RedisConnectionFactory factory = this.getConnectionFactory();
RedisConnection conn = null;
Object var11;
try {
if (this.enableTransactionSupport) {
conn = RedisConnectionUtils.bindConnection(factory, this.enableTransactionSupport);
} else {
conn = RedisConnectionUtils.getConnection(factory);
}
boolean existingConnection = TransactionSynchronizationManager.hasResource(factory);
RedisConnection connToUse = this.preProcessConnection(conn, existingConnection);
boolean pipelineStatus = connToUse.isPipelined();
if (pipeline && !pipelineStatus) {
connToUse.openPipeline();
}
RedisConnection connToExpose = exposeConnection ? connToUse : this.createRedisConnectionProxy(connToUse);
T result = action.doInRedis(connToExpose);
if (pipeline && !pipelineStatus) {
connToUse.closePipeline();
}
var11 = this.postProcessResult(result, connToUse, existingConnection);
} finally {
RedisConnectionUtils.releaseConnection(conn, factory);
}
return var11;
}
由于Azure redis缓存主备切换无法避免,为了保证这个情况不对服务造成影响甚至中断,需要在redis抛出RedisConnectionFailureException|JedisConnectionException
异常时,重试抛出异常的操作。
基本方案,引入spring-retry框架,对redis操作进行重试。
为了保证redis操作的性能,否决对execute
方法加AOP
方案
使用方案2时,开始方案如下:
public class HapRedisTemplate<K, V> extends RedisTemplate<K, V> {
private static Logger LOG = LoggerFactory.getLogger(HapRedisTemplate.class);
@Override
public <T> T execute(RedisCallback<T> action, boolean exposeConnection, boolean pipeline) {
T value = null;
try {
value = super.execute(action, exposeConnection, pipeline);
} catch (JedisConnectionException | RedisConnectionFailureException e) {
LOG.error("OverrideHapRedisTemplate {}-{}", e.getMessage(), e);
RetryTemplate retryTemplate = RetryTemplateUtil.getSimpleRetryTemplate(3, 200L);
try {
value = retryTemplate.execute((RetryCallback<T, Throwable>) (context) ->
super.execute(action, exposeConnection, pipeline));
} catch (Throwable t) {
LOG.error("OverrideHapRedisTemplate {}-{}", t.getMessage(), t);
}
}
return value;
}
}
对RedisTemplate类中的execute方法抛出的异常拦截,并重新调用三次(获取spring-retry的RetryTemplate, 由于这个类是非线程安全的,就没有交给spring容器,懒得配置bean。成功拿到value则停止retry; 三次结束还是失败则返回null ),每次间隔200ml, 发现不能解决问题,因为重新执行RedisTemplate中的execute方法拿到的redis连接还是从失效的那个连接池里面拿的,不能保证有效,甚至在主备切换的这个时间内,一定是无效的。
由于生产环境连接池配置不能打开testOnBorrow,同时又要保证可用性,所以对初始方案做了一点优化,在execute方案抛出异常时,重试execute方法时,保证重试过程中拿到的连接一定是有效的.
代码如下:
public class HapRedisTemplate<K, V> extends RedisTemplate<K, V> {
private static Logger LOG = LoggerFactory.getLogger(HapRedisTemplate.class);
private RedisTemplate<K, V> backupRedisTemplate;
@Override
public <T> T execute(RedisCallback<T> action, boolean exposeConnection, boolean pipeline) {
T value = null;
try {
value = super.execute(action, exposeConnection, pipeline);
} catch (JedisConnectionException | RedisConnectionFailureException e) {
LOG.error("OverrideHapRedisTemplate {}-{}", e.getMessage(), e);
RetryTemplate retryTemplate = RetryTemplateUtil.getSimpleRetryTemplate(3, 200L);
try {
value = retryTemplate.execute((RetryCallback<T, Throwable>) (context) ->
backupRedisTemplate.execute(action, exposeConnection, pipeline));
} catch (Throwable t) {
LOG.error("OverrideHapRedisTemplate {}-{}", t.getMessage(), t);
}
}
return value;
}
public RedisTemplate<K, V> getBackupRedisTemplate() {
return backupRedisTemplate;
}
public void setBackupRedisTemplate(RedisTemplate<K, V> backupRedisTemplate) {
this.backupRedisTemplate = backupRedisTemplate;
}
}
基本思路不变,还是拦截redis抛出的异常,只不过在catch块中不再调用本实例的execute方法,而是通过备用redisTemplate实例执行execute方法,而备用redisTemplate引用的连接池呢,打开testOnBorrow开关。
最终的spring配置如下:
<bean id="v2redisConnectionFactory" class="com.hand.hap.core.JedisConnectionFactoryBean">
<property name="useSentinel" value="${redis.useSentinel}"/>
<property name="sentinelConfiguration" ref="redisSentinelConfiguration"/>
<property name="hostName" value="${REDIS_IP}"/>
<property name="port" value="${REDIS_PORT}"/>
<property name="database" value="${REDIS_DB}"/>
<property name="password" value="${REDIS_PASSWORD}"/>
<property name="poolConfig">
<bean class="redis.clients.jedis.JedisPoolConfig">
<property name="maxIdle" value="999"/>
<property name="maxTotal" value="9999"/>
<property name="minIdle" value="20"/>
bean>
property>
bean>
<bean id="v2redisTemplate" name="redisTemplate,v2redisTemplate" class="com.hand.hap.cache.impl.HapRedisTemplate">
<property name="backupRedisTemplate" ref="backupV2RedisTemplate"/>
<property name="connectionFactory" ref="v2redisConnectionFactory"/>
<property name="keySerializer" ref="stringRedisSerializer"/>
<property name="valueSerializer" ref="stringRedisSerializer"/>
<property name="hashKeySerializer" ref="stringRedisSerializer"/>
<property name="hashValueSerializer" ref="stringRedisSerializer"/>
bean>
<bean id="backupV2RedisTemplate" class="org.springframework.data.redis.core.RedisTemplate">
<property name="connectionFactory" ref="backupRedisConnectFactory"/>
<property name="keySerializer" ref="stringRedisSerializer"/>
<property name="valueSerializer" ref="stringRedisSerializer"/>
<property name="hashKeySerializer" ref="stringRedisSerializer"/>
<property name="hashValueSerializer" ref="stringRedisSerializer"/>
bean>
<bean id="backupRedisConnectFactory" class="com.hand.hap.core.JedisConnectionFactoryBean">
<property name="useSentinel" value="${redis.useSentinel}"/>
<property name="sentinelConfiguration" ref="redisSentinelConfiguration"/>
<property name="hostName" value="${REDIS_IP}"/>
<property name="port" value="${REDIS_PORT}"/>
<property name="database" value="${REDIS_DB}"/>
<property name="password" value="${REDIS_PASSWORD}"/>
<property name="poolConfig" ref="backupPoolConfig"/>
bean>
<bean id="backupPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
<property name="maxIdle" value="20"/>
<property name="maxTotal" value="50"/>
<property name="minIdle" value="10"/>
<property name="testOnBorrow" value="true"/>
<property name="testWhileIdle" value="true"/>
bean>
由于redis主备切换完成后,redis连接池中的连接会重新建立,这样就能保证系统即能保证性能又能保证容错了。