故障发现原理
基本原理
- 节点之间通过
ping/pong
实现故障发现,不需要 sentinel;
主观下线
- 某个节点认为另一个节点不可用,是一种“偏见”;
客观下线
- 当半数以上持有槽的主节点都标记某节点主观下线;
- 会向集群中的所有节点发通知,让他们把客观下线的节点标记为客观下线;
- 然后准备客观下线的节点的从节点启动故障转移的流程;
故障恢复原理
资格检查
- 每个从节点检查与故障主节点的断线时间;
- 如果断线时间超过
cluster-node-timeout(15000)
*cluster-slave-validity-factor(10)
取消转为主节点的资格;
准备选举时间
- 偏移量最小的 slave 会得到最短的延迟选举时间,是为了保证该 slave 有更多的票数成为 master;
选举投票
- slave 的延迟选举时间到了之后,会开始被其他活着的 master 投票,所以偏移量最大的 slave 最先开始被投票,所以获得的票数就更多,更有可能成为 master;
- slave 的票数到达整个集群节点数量
1/2 + 1
后,就可以成为主节点;
替换主节点
- 当前 slave 取消复制变为 master,
slaveof no one
; - 执行
clusterDelSlot
撤销故障 master 负责的槽,并执行clusterAddSlot
把这些槽分配给自己; - 向集群中广播自己的 pong 消息,表明已经替换了故障从节点;
故障模拟
查看集群状态
[root@redis3_0_7 config]# redis-trib.rb info 127.0.0.1:7000
127.0.0.1:7000 (7d2976bc...) -> 2 keys | 5461 slots | 1 slaves.
127.0.0.1:7002 (e956a35a...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:7001 (d5dc769a...) -> 2 keys | 5462 slots | 1 slaves.
[OK] 4 keys in 3 masters.
0.00 keys per slot on average.
启动客户端测试程序
package com.example.redis.client.cluster;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.JedisCluster;
import java.util.concurrent.TimeUnit;
@Service
public class JedisClusterFailoverTest implements Runnable{
private static final Logger logger = LoggerFactory.getLogger(JedisClusterFailoverTest.class);
public volatile boolean signal = true;
@Autowired
private JedisCluster jedisCluster;
@Override
public void run() {
int counter = 0;
while (signal) {
int num = counter ++;
try {
String key = "k-" + num;
String value = "v-" + num;
jedisCluster.set(key, value);
logger.info("{} value is {}", key, jedisCluster.get(key));
TimeUnit.SECONDS.sleep(1);
} catch (Exception e) {
logger.error(e.getMessage(), e);
} finally {
}
}
}
}
查看 7000 的 process_id,准备 kill 掉
[root@redis3_0_7 ~]# redis-cli -p 7000 info server | grep process_id
process_id:30344
kill 掉 7000
[root@redis3_0_7 ~]# kill -9 30344
查看客户端日志
- 中断了大概 17s,7000 的 slave 7003 晋升为 master;
[root@redis3_0_7 ~]# cat JedisClusterFailoverTest.log
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.5.RELEASE)
2019-05-19 16:03:47.266 INFO 171480 --- [ main] c.e.redis.client.ClientApplication : Starting ClientApplication on LiXinlei-RSC with PID 171480 (D:\Project\MOOC\coding-151\client\target\classes started by LiXinlei in D:\Project\MOOC\coding-151\client)
2019-05-19 16:03:47.270 INFO 171480 --- [ main] c.e.redis.client.ClientApplication : No active profile set, falling back to default profiles: default
2019-05-19 16:03:48.733 INFO 171480 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2019-05-19 16:03:48.763 INFO 171480 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2019-05-19 16:03:48.763 INFO 171480 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.19]
2019-05-19 16:03:48.967 INFO 171480 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2019-05-19 16:03:48.967 INFO 171480 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 1628 ms
2019-05-19 16:03:49.308 INFO 171480 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2019-05-19 16:03:49.588 INFO 171480 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2019-05-19 16:03:49.592 INFO 171480 --- [ main] c.e.redis.client.ClientApplication : Started ClientApplication in 3.009 seconds (JVM running for 6.442)
2019-05-19 16:03:59.670 INFO 171480 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2019-05-19 16:03:59.671 INFO 171480 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2019-05-19 16:03:59.680 INFO 171480 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 9 ms
2019-05-19 16:03:59.720 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-0 value is v-0
2019-05-19 16:04:00.724 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-1 value is v-1
2019-05-19 16:04:01.730 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-2 value is v-2
2019-05-19 16:04:02.736 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-3 value is v-3
2019-05-19 16:04:03.741 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-4 value is v-4
2019-05-19 16:04:04.744 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-5 value is v-5
2019-05-19 16:04:05.749 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-6 value is v-6
2019-05-19 16:04:06.752 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-7 value is v-7
2019-05-19 16:04:07.755 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-8 value is v-8
2019-05-19 16:04:08.757 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-9 value is v-9
2019-05-19 16:04:09.804 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:09.853 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:09.857 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-12 value is v-12
2019-05-19 16:04:10.860 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-13 value is v-13
2019-05-19 16:04:11.905 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:11.942 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:11.945 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-16 value is v-16
2019-05-19 16:04:12.948 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-17 value is v-17
2019-05-19 16:04:14.002 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:14.062 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:14.066 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-20 value is v-20
2019-05-19 16:04:15.113 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:15.116 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-22 value is v-22
2019-05-19 16:04:16.119 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-23 value is v-23
2019-05-19 16:04:17.124 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-24 value is v-24
2019-05-19 16:04:18.175 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:18.178 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-26 value is v-26
2019-05-19 16:04:19.181 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-27 value is v-27
2019-05-19 16:04:20.185 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-28 value is v-28
2019-05-19 16:04:21.248 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:21.251 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-30 value is v-30
2019-05-19 16:04:22.255 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-31 value is v-31
2019-05-19 16:04:23.291 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:23.294 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-33 value is v-33
2019-05-19 16:04:24.298 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-34 value is v-34
2019-05-19 16:04:25.301 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-35 value is v-35
2019-05-19 16:04:26.341 ERROR 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : Unexpected end of stream.
redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:199) ~[jedis-2.9.0.jar:na]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:239) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.Jedis.set(Jedis.java:121) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:101) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster$1.execute(JedisCluster.java:98) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:31) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisCluster.set(JedisCluster.java:103) ~[jedis-2.9.0.jar:na]
at com.example.redis.client.cluster.JedisClusterFailoverTest.run(JedisClusterFailoverTest.java:29) ~[classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2019-05-19 16:04:26.344 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-37 value is v-37
2019-05-19 16:04:27.347 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-38 value is v-38
2019-05-19 16:04:28.351 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-39 value is v-39
2019-05-19 16:04:29.354 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-40 value is v-40
2019-05-19 16:04:30.357 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-41 value is v-41
2019-05-19 16:04:31.360 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-42 value is v-42
2019-05-19 16:04:32.362 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-43 value is v-43
2019-05-19 16:04:33.367 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-44 value is v-44
2019-05-19 16:04:34.371 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-45 value is v-45
2019-05-19 16:04:35.375 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-46 value is v-46
2019-05-19 16:04:36.378 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-47 value is v-47
2019-05-19 16:04:37.381 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-48 value is v-48
2019-05-19 16:04:38.383 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-49 value is v-49
2019-05-19 16:04:39.386 INFO 171480 --- [pool-1-thread-1] c.e.r.c.c.JedisClusterFailoverTest : k-50 value is v-50
查看集群状态
- 7003 成为主节点;
[root@redis3_0_7 ~]# redis-trib.rb info 127.0.0.1:7001
127.0.0.1:7001 (d5dc769a...) -> 865 keys | 5462 slots | 1 slaves.
127.0.0.1:7003 (e3c69ce8...) -> 823 keys | 5461 slots | 0 slaves.
127.0.0.1:7002 (e956a35a...) -> 839 keys | 5461 slots | 1 slaves.
[OK] 2527 keys in 3 masters.
0.15 keys per slot on average.
重新启动 7000
- 7000 变成 7003 的 slave;
[root@redis3_0_7 config]# redis-server redis-7000.conf
[root@redis3_0_7 config]# redis-cli -p 7005 cluster slots
1) 1) (integer) 5461
2) (integer) 10922
3) 1) "127.0.0.1"
2) (integer) 7001
4) 1) "127.0.0.1"
2) (integer) 7004
2) 1) (integer) 10923
2) (integer) 16383
3) 1) "127.0.0.1"
2) (integer) 7002
4) 1) "127.0.0.1"
2) (integer) 7005
3) 1) (integer) 0
2) (integer) 5460
3) 1) "127.0.0.1"
2) (integer) 7003
4) 1) "127.0.0.1"
2) (integer) 7000