故障转移

故障发现原理

基本原理
  • 节点之间通过 ping/pong 实现故障发现,不需要 sentinel;
主观下线
  • 某个节点认为另一个节点不可用,是一种“偏见”;
客观下线
  • 当半数以上持有槽的主节点都标记某节点主观下线;
  • 会向集群中的所有节点发通知,让他们把客观下线的节点标记为客观下线;
  • 然后准备客观下线的节点的从节点启动故障转移的流程;

故障恢复原理

资格检查
  • 每个从节点检查与故障主节点的断线时间;
  • 如果断线时间超过 cluster-node-timeout(15000) * cluster-slave-validity-factor(10) 取消转为主节点的资格;
准备选举时间
  • 偏移量最小的 slave 会得到最短的延迟选举时间,是为了保证该 slave 有更多的票数成为 master;
选举投票
  • slave 的延迟选举时间到了之后,会开始被其他活着的 master 投票,所以偏移量最大的 slave 最先开始被投票,所以获得的票数就更多,更有可能成为 master;
  • slave 的票数到达整个集群节点数量 1/2 + 1 后,就可以成为主节点;
替换主节点
  • 当前 slave 取消复制变为 master,slaveof no one
  • 执行 clusterDelSlot 撤销故障 master 负责的槽,并执行 clusterAddSlot 把这些槽分配给自己;
  • 向集群中广播自己的 pong 消息,表明已经替换了故障从节点;

故障模拟

查看集群状态
[root@redis3_0_7 config]# redis-trib.rb info 127.0.0.1:7000
127.0.0.1:7000 (7d2976bc...) -> 2 keys | 5461 slots | 1 slaves.
127.0.0.1:7002 (e956a35a...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:7001 (d5dc769a...) -> 2 keys | 5462 slots | 1 slaves.
[OK] 4 keys in 3 masters.
0.00 keys per slot on average.
启动客户端测试程序
package com.example.redis.client.cluster;


import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.JedisCluster;

import java.util.concurrent.TimeUnit;

@Service
public class JedisClusterFailoverTest implements Runnable{

    private static final Logger logger = LoggerFactory.getLogger(JedisClusterFailoverTest.class);
    public volatile boolean signal = true;

    @Autowired
    private JedisCluster jedisCluster;

    @Override
    public void run() {
        int counter = 0;
        while (signal) {
            int num = counter ++;
            try {
                String key = "k-" + num;
                String value = "v-" + num;
                jedisCluster.set(key, value);
                logger.info("{} value is {}", key, jedisCluster.get(key));
                TimeUnit.SECONDS.sleep(1);
            } catch (Exception e) {
                logger.error(e.getMessage(), e);
            } finally {

            }
        }
    }

}
查看 7000 的 process_id,准备 kill 掉
[root@redis3_0_7 ~]# redis-cli -p 7000 info server | grep process_id
process_id:30344
kill 掉 7000
[root@redis3_0_7 ~]# kill -9 30344
查看客户端日志
  • 中断了大概 17s,7000 的 slave 7003 晋升为 master;
[root@redis3_0_7 ~]# cat JedisClusterFailoverTest.log 

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.5.RELEASE)

2019-05-19 16:03:47.266  INFO 171480 --- [           main] c.e.redis.client.ClientApplication       : Starting ClientApplication on LiXinlei-RSC with PID 171480 (D:\Project\MOOC\coding-151\client\target\classes started by LiXinlei in D:\Project\MOOC\coding-151\client)
2019-05-19 16:03:47.270  INFO 171480 --- [           main] c.e.redis.client.ClientApplication       : No active profile set, falling back to default profiles: default
2019-05-19 16:03:48.733  INFO 171480 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2019-05-19 16:03:48.763  INFO 171480 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2019-05-19 16:03:48.763  INFO 171480 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.19]
2019-05-19 16:03:48.967  INFO 171480 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2019-05-19 16:03:48.967  INFO 171480 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 1628 ms
2019-05-19 16:03:49.308  INFO 171480 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2019-05-19 16:03:49.588  INFO 171480 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2019-05-19 16:03:49.592  INFO 171480 --- [           main] c.e.redis.client.ClientApplication       : Started ClientApplication in 3.009 seconds (JVM running for 6.442)
2019-05-19 16:03:59.670  INFO 171480 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2019-05-19 16:03:59.671  INFO 171480 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2019-05-19 16:03:59.680  INFO 171480 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 9 ms
2019-05-19 16:03:59.720  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-0 value is v-0
2019-05-19 16:04:00.724  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-1 value is v-1
2019-05-19 16:04:01.730  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-2 value is v-2
2019-05-19 16:04:02.736  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-3 value is v-3
2019-05-19 16:04:03.741  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-4 value is v-4
2019-05-19 16:04:04.744  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-5 value is v-5
2019-05-19 16:04:05.749  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-6 value is v-6
2019-05-19 16:04:06.752  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-7 value is v-7
2019-05-19 16:04:07.755  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-8 value is v-8
2019-05-19 16:04:08.757  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-9 value is v-9
2019-05-19 16:04:09.804 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:09.853 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:09.857  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-12 value is v-12
2019-05-19 16:04:10.860  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-13 value is v-13
2019-05-19 16:04:11.905 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:11.942 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:11.945  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-16 value is v-16
2019-05-19 16:04:12.948  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-17 value is v-17
2019-05-19 16:04:14.002 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:14.062 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:14.066  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-20 value is v-20
2019-05-19 16:04:15.113 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:15.116  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-22 value is v-22
2019-05-19 16:04:16.119  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-23 value is v-23
2019-05-19 16:04:17.124  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-24 value is v-24
2019-05-19 16:04:18.175 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:18.178  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-26 value is v-26
2019-05-19 16:04:19.181  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-27 value is v-27
2019-05-19 16:04:20.185  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-28 value is v-28
2019-05-19 16:04:21.248 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:21.251  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-30 value is v-30
2019-05-19 16:04:22.255  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-31 value is v-31
2019-05-19 16:04:23.291 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:23.294  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-33 value is v-33
2019-05-19 16:04:24.298  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-34 value is v-34
2019-05-19 16:04:25.301  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-35 value is v-35
2019-05-19 16:04:26.341 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : Unexpected end of stream.

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
    at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
    at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

2019-05-19 16:04:26.344  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-37 value is v-37
2019-05-19 16:04:27.347  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-38 value is v-38
2019-05-19 16:04:28.351  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-39 value is v-39
2019-05-19 16:04:29.354  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-40 value is v-40
2019-05-19 16:04:30.357  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-41 value is v-41
2019-05-19 16:04:31.360  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-42 value is v-42
2019-05-19 16:04:32.362  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-43 value is v-43
2019-05-19 16:04:33.367  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-44 value is v-44
2019-05-19 16:04:34.371  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-45 value is v-45
2019-05-19 16:04:35.375  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-46 value is v-46
2019-05-19 16:04:36.378  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-47 value is v-47
2019-05-19 16:04:37.381  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-48 value is v-48
2019-05-19 16:04:38.383  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-49 value is v-49
2019-05-19 16:04:39.386  INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest       : k-50 value is v-50
查看集群状态
  • 7003 成为主节点;
[root@redis3_0_7 ~]# redis-trib.rb info 127.0.0.1:7001
127.0.0.1:7001 (d5dc769a...) -> 865 keys | 5462 slots | 1 slaves.
127.0.0.1:7003 (e3c69ce8...) -> 823 keys | 5461 slots | 0 slaves.
127.0.0.1:7002 (e956a35a...) -> 839 keys | 5461 slots | 1 slaves.
[OK] 2527 keys in 3 masters.
0.15 keys per slot on average.
重新启动 7000
  • 7000 变成 7003 的 slave;
[root@redis3_0_7 config]# redis-server redis-7000.conf
[root@redis3_0_7 config]# redis-cli -p 7005 cluster slots
1) 1) (integer) 5461
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 7001
   4) 1) "127.0.0.1"
      2) (integer) 7004
2) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7002
   4) 1) "127.0.0.1"
      2) (integer) 7005
3) 1) (integer) 0
   2) (integer) 5460
   3) 1) "127.0.0.1"
      2) (integer) 7003
   4) 1) "127.0.0.1"
      2) (integer) 7000

你可能感兴趣的:(故障转移)