线上故障之Redis缓存数据异常

目录

故障现象

问题结论分析

问题优化

1、redis操作优化

2、业务降级容错

3、监控层面

---

故障现象

Redis online_user_{id}缓存数据异常问题

redis存在异常数据

{"queueList":["10000993","10000994","10000995"],"agent":{"queueList":["10000993","10000994","10000995"],"webrtcSipKey":"wsIFFl0d7C","departmentConfig":{"departmentIdList":[],"leaderId":""},"roles":["6114cd553f09bc0011563d17","6125b15cc019c600111524ba","6125b27878aefa001126f3c1"],"mobile":"","agentName":"孙李锁","updateUser":"62a9482305695f65e4b3a8d1","toAnswerOffline":"0","updateTime":"2022-06-15 10:47:15","agentNumber":"2020","accountId":"2000687","crmId":"","lastLoginTime":"","sipNumber":"","lockTime":"2022-04-04 12:01:59","createTime":"2021-07-20 14:47:10","agentVersion":"L00016","createUser":"60f66f31f912040011401940","_id":"60f6716ebc28d30011b90372","webrtcSipNumber":"2000687w2020","userIcon":"","email":"","status":"enable"},"pushServerPort":3000,"webrtcSipKey":"wsIFFl0d7C","departmentConfig":{"departmentIdList":[],"leaderId":""},"roles":["6114cd553f09bc0011563d17","6125b15cc019c600111524ba","6125b27878aefa001126f3c1"],"toAnswerOffline":"0","connTime":"2022-06-15 13:06:39","socketId":"fbSTWB_akZVJX7XxABh3","pushServerIp":"10.42.14.181","sipNumber":"","lockTime":"2022-04-04 12:01:59","email":"","mobile":"","agentName":"孙李锁","updateUser":"60f67199bc28d30011b903ec","updateTime":"2022-06-15 13:57:03","agentNumber":"2020","accountId":"2000687","crmId":"","lastLoginTime":"","createTime":"2021-07-20 14:47:10","agentVersion":"L00016","createUser":"60f66f31f912040011401940","_id":"60f6716ebc28d30011b90372","webrtc*3 $3 SET $41 cache_onlineuser_6257c85d6f21d7001150784"}

可以看到在正常的json数据后面多了一些没用的数据，看着是redis的另一个key的信息

问题结论分析

原因：cache_onlineuser_6257c85d6f21d7001150784 这个缓存数据存在异常，导致业务解析失败

经排查代码里使用了自己写的RedisAPI工具类，且多个线程公用了同一个redisAPI实例导致的

        CompletableFuture.allOf(
                keySet.stream().map(
                        v -> {
                            JSONObject userCacheInfo = JSONObject.parseObject(onlineUsers.get(v));
                            JSONObject userJson = WebUtils.toJsonObject(agents.get(Integer.parseInt(v)));
                            userJson.remove("__v");
                            userJson.remove("hashPassword");
                            userCacheInfo.putAll(userJson);
                            String key = RedisConstant.REDIS_PREFIX_CACHE_ONLINEUSER + agents.get(Integer.parseInt(v)).get_id();
                            return CompletableFuture.runAsync(() -> redis.set(key, userCacheInfo.toJSONString()))

                        }).toArray(CompletableFuture[]::new)

        ).join();

经确认，redis的这个工具类是非线程安全的（jedis本身也是线程不安全的），表现为在set, get方法在多线程调用时会出现数据错乱。jedis发送命令和获取返回值时使用全局变量RedisOutputStream和RedisInputStream，所以会导致数据偶发的异常

问题优化

1、redis操作优化

• 优化此处逻辑，存online_user_{id} redis key的时候，避免多线程使用同一个Redis实例。避免掉异常数据。
• 类似代码逻辑调整。所有使用Redis包的地方，排查此类问题并修复

        方案1：改为使用不同的redis实例

        方案2：使用批量操作mset, mget等优化替代多线程操作redis

优化后的代码：

      Set keySet = onlineUsers.keySet();
        keySet.forEach(v -> {
            JSONObject userCacheInfo = JSONObject.parseObject(onlineUsers.get(v));
            JSONObject userJson = WebUtils.toJsonObject(agents.get(Integer.parseInt(v)));
            userJson.remove("__v");
            userJson.remove("hashPassword");
            userCacheInfo.putAll(userJson);
            String key = RedisConstant.REDIS_PREFIX_CACHE_ONLINEUSER + agents.get(Integer.parseInt(v)).get_id();
            redis.set(key, userCacheInfo.toJSONString());
        });

2、业务降级容错

• 针对所有从redis取出的数据，使用try catch校验数据和容错

3、监控层面

• 添加关键redis数据解析失败的告警通知