分布式架构高可用架构篇_01_zookeeper集群的安装、配置、高可用测试

Dubbo 建议使用 Zookeeper 作为服务的注册中心 

Zookeeper 集群中只要有过半的节点是正常的情况下，那么整个集群对外就是可用的。正是基于这个 特性，要将 ZK 集群的节点数量要为奇数（2n+1：如 3、5、7 个节点）较为合适。 

ZooKeeper 与 Dubbo 服务集群架构图

服务器 1：192.168.2.81 端口：2181、2881、3881

 

服务器 2：192.168.2.82 端口：2182、2882、3882

 

服务器 3：192.168.2.83 端口：2183、2883、3883

 

 

1、 修改操作系统的/etc/hosts 文件，添加 IP 与主机名映射：

 

# zookeeper cluster servers 

192.168.2.81 edu-zk-01 

192.168.2.82 edu-zk-02 

192.168.2.83 edu-zk-03

[root@edu-zk-01 root]# cat /etc/hosts | grep edu
192.168.2.81   edu-zk-01
[root@edu-zk-01 root]# 

[root@edu-zk-02 ~]# cat /etc/hosts | grep edu
192.168.2.82 edu-zk-02
[root@edu-zk-02 ~]#

2、 下载或上传 zookeeper-3.4.6.tar.gz 到/home/yxq/zookeeper 目录

[root@edu-zk-01 root]# clear
[root@edu-zk-01 root]# cd /home/yxq/zookeeper
[root@edu-zk-01 zookeeper]# wget http://apache.fayea.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

[root@edu-zk-02 ~]# mkdir /home/yxq/zookeeper
[root@edu-zk-02 ~]# cd /home/yxq/zookeeper/
[root@edu-zk-02 zookeeper]# wget http://apache.fayea.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

3、 解压 zookeeper 安装包，并按节点号对 zookeeper 目录重命名： 

服务器1

[root@edu-zk-01 zookeeper]# tar -zxvf zookeeper-3.4.6.tar.gz 

[root@edu-zk-01 zookeeper]# mv zookeeper-3.4.6 node-01

服务器2:

[root@edu-zk-02 zookeeper]# tar -zxvf zookeeper-3.4.6.tar.gz

[root@edu-zk-02 zookeeper]# mv zookeeper-3.4.6 node-02

服务器3:

[root@edu-zk-03 ~]# cd /home/yxq/zookeeper
[root@edu-zk-03 zookeeper]# wget http://apache.fayea.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

[root@edu-zk-03 zookeeper]# tar -zxvf zookeeper-3.4.6.tar.gz 

[root@edu-zk-03 zookeeper]# mv zookeeper-3.4.6 node-03

4、 在各 zookeeper 节点目录下创建以下目录：

[root@edu-zk-01 zookeeper]# cd node-01/
[root@edu-zk-01 node-01]# mkdir data
[root@edu-zk-01 node-01]# mkdir logs

[root@edu-zk-02 zookeeper]# cd node-02/
[root@edu-zk-02 node-02]# mkdir data
[root@edu-zk-02 node-02]# mkdir logs

[root@edu-zk-03 zookeeper]# cd node-03/
[root@edu-zk-03 node-03]# mkdir data
[root@edu-zk-03 node-03]# mkdir logs

 将 zookeeper/node-0X/conf 目录下的 zoo_sample.cfg 文件拷贝一份，命名为 zoo.cfg

[root@edu-zk-01 node-01]# cd conf
[root@edu-zk-01 conf]# cp zoo_sample.cfg  zoo.cfg

[root@edu-zk-02 node-02]# cd conf
[root@edu-zk-02 conf]# cp zoo_sample.cfg  zoo.cfg

[root@edu-zk-03 node-03]# cd conf
[root@edu-zk-03 conf]# cp zoo_sample.cfg zoo.cfg

6、 修改 zoo.cfg 配置文件：

zookeeper/node-01 的配置（/home/yxq/zookeeper/node-01/conf/zoo.cfg）如下： 

tickTime=2000 

initLimit=10 

syncLimit=5 

dataDir=/home/yxq/zookeeper/node-01/data

 dataLogDir=/home/yxq/zookeeper/node-01/logs 

clientPort=2181 

server.1=192.168.2.81:2881:3881 

server.2=192.168.2.82:2882:3882 

server.3=192.168.2.83:2883:3883

zookeeper/node-02的配置（/home/yxq/zookeeper/node-02/conf/zoo.cfg）如下： 

tickTime=2000 

initLimit=10 

syncLimit=5 

dataDir=/home/yxq/zookeeper/node-02/data

 dataLogDir=/home/yxq/zookeeper/node-02/logs 

clientPort=2181 

server.1=192.168.2.81:2881:3881 

server.2=192.168.2.82:2882:3882 

server.3=192.168.2.83:2883:3883

zookeeper/node-03 的配置（/home/yxq/zookeeper/node-03/conf/zoo.cfg）如下： 

tickTime=2000 

initLimit=10 

syncLimit=5 

dataDir=/home/yxq/zookeeper/node-03/data

 dataLogDir=/home/yxq/zookeeper/node-03/logs 

clientPort=2181 

server.1=192.168.2.81:2881:3881 

server.2=192.168.2.82:2882:3882 

server.3=192.168.2.83:2883:3883

参数说明:

 tickTime=2000 

tickTime 这个时间是作为 Zookeeper 服务器之间或客户端与服务器之间维持心跳的时间间隔,也就是每

 个 tickTime 时间就会发送一个心跳。 

initLimit=10 

initLimit 这个配置项是用来配置 Zookeeper 接受客户端（这里所说的客户端不是用户连接 Zookeeper

 服务器的客户端,而是 Zookeeper 服务器集群中连接到 Leader 的 Follower 服务器）初始化连接时最长

 能忍受多少个心跳时间间隔数。当已经超过 10 个心跳的时间（也就是 tickTime）长度后 Zookeeper 服

 务器还没有收到客户端的返回信息,那么表明这个客户端连接失败。总的时间长度就是 10*2000=20 秒。

 syncLimit=5

 syncLimit 这个配置项标识 Leader 与 Follower 之间发送消息,请求和应答时间长度,最长不能超过多少

 个 tickTime 的时间长度,总的时间长度就是 5*2000=10 秒。

 dataDir=/home/wusc/zookeeper/node-01/data

 dataDir 顾名思义就是 Zookeeper 保存数据的目录,默认情况下 Zookeeper 将写数据的日志文件也保存在 这个目录里。

clientPort=2181

 clientPort 这个端口就是客户端（应用程序）连接 Zookeeper 服务器的端口,Zookeeper 会监听这个端

 server.A=B：C：D

 server.1=192.168.2.81:2881:3881

 server.2=192.168.2.82:2882:3882

 server.3=192.168.2.83:2883:3883

 A 是一个数字,表示这个是第几号服务器；

 B 是这个服务器的 IP 地址（或者是与 IP 地址做了映射的主机名）；

 C 第一个端口用来集群成员的信息交换,表示这个服务器与集群中的 Leader 服务器交换信息的端口；

 D 是在 leader 挂掉时专门用来进行选举 leader 所用的端口。

 注意：如果是伪集群的配置方式，不同的 Zookeeper 实例通信端口号不能一样，所以要给它们分配不 同的端口号。

 

7、 在 dataDir=/home/wusc/zookeeper/node-0X/data 下创建 myid 文件

编辑 myid 文件，并在对应的 IP 的机器上输入对应的编号。如在 node-01 上，myid 文件内容就是 1,node-02 上就是 2，node-03 上就是 3：

[root@edu-zk-01 node-01]# cd data
[root@edu-zk-01 data]# vi myid
[root@edu-zk-01 data]# cat myid 
1
[root@edu-zk-01 data]# 

[root@edu-zk-02 conf]# cd ..
[root@edu-zk-02 node-02]# cd data
[root@edu-zk-02 data]# vi myid
[root@edu-zk-02 data]# cat myid 
2

[root@edu-zk-03 conf]# cd ..
[root@edu-zk-03 node-03]# cd data
[root@edu-zk-03 data]# vi myid
[root@edu-zk-03 data]# cat myid 
3
[root@edu-zk-03 data]# 

8、 在防火墙中打开要用到的端口 218X、288X、388X 

切换到 root 用户权限，执行以下命令：

[root@edu-zk-01 data]# vi /etc/sysconfig/iptables
[root@edu-zk-01 data]# service iptables restart

-A INPUT -m state --state NEW -m tcp -p tcp --dport 2181 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 2881 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 3881 -j ACCEPT
edu-zk-03,edu-zk-03同理

[root@edu-zk-02 data]# vi /etc/sysconfig/iptables
[root@edu-zk-02 data]# service iptables restart

-A INPUT -m state --state NEW -m tcp -p tcp --dport 2182-j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 2882 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 3882 -j ACCEPT

[root@edu-zk-03 data]# vi /etc/sysconfig/iptables
[root@edu-zk-03 data]# service iptables restart

-A INPUT -m state --state NEW -m tcp -p tcp --dport 2183 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 2883 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 3883 -j ACCEPT

9、 启动并测试 zookeeper（要用 yxq用户启动，不要用 root）:

先启用edu-zk-01.集群环境下，只有过半的节点运行正常才不报错，这里报错是正常的

[root@edu-zk-01 root]# clear
[root@edu-zk-01 root]# who
yxq      pts/0        2016-04-15 00:48 (192.168.2.61)
[root@edu-zk-01 root]# /home/yxq/zookeeper/node-01/bin/zkServer.sh start
JMX enabled by default
Using config: /home/yxq/zookeeper/node-01/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

查看是否正常启动

其中，QuorumPeerMain 是 zookeeper 进程，说明启动正常

[root@edu-zk-01 root]# jps
3293 Jps
3240 QuorumPeerMain
[root@edu-zk-01 root]# tail -500f zookeeper.out 
2016-04-15 01:11:57,970 [myid:] - INFO  [main:QuorumPeerConfig@103] - Reading configuration from: /home/yxq/zookeeper/node-01/bin/../conf/zoo.cfg
2016-04-15 01:11:57,974 [myid:] - INFO  [main:QuorumPeerConfig@340] - Defaulting to majority quorums
2016-04-15 01:11:57,976 [myid:1] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-04-15 01:11:57,976 [myid:1] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-04-15 01:11:57,977 [myid:1] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-04-15 01:11:57,985 [myid:1] - INFO  [main:QuorumPeerMain@127] - Starting quorum peer
2016-04-15 01:11:57,993 [myid:1] - INFO  [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
2016-04-15 01:11:58,015 [myid:1] - INFO  [main:QuorumPeer@959] - tickTime set to 2000
2016-04-15 01:11:58,015 [myid:1] - INFO  [main:QuorumPeer@979] - minSessionTimeout set to -1
2016-04-15 01:11:58,015 [myid:1] - INFO  [main:QuorumPeer@990] - maxSessionTimeout set to -1
2016-04-15 01:11:58,015 [myid:1] - INFO  [main:QuorumPeer@1005] - initLimit set to 10
2016-04-15 01:11:58,028 [myid:1] - INFO  [main:FileSnap@83] - Reading snapshot /home/yxq/zookeeper/node-01/data/version-2/snapshot.0
2016-04-15 01:11:58,037 [myid:1] - INFO  [Thread-1:QuorumCnxManager$Listener@504] - My election bind port: /192.168.2.81:3881
2016-04-15 01:11:58,044 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer@714] - LOOKING
2016-04-15 01:11:58,045 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@815] - New election. My id =  1, proposed zxid=0x0
2016-04-15 01:11:58,047 [myid:1] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state)
2016-04-15 01:11:58,050 [myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channel to 2 at election address /192.168.2.82:3882
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:744)
2016-04-15 01:11:58,053 [myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channel to 3 at election address /192.168.2.83:3883
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:744)
2016-04-15 01:11:58,256 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 2 at election address /192.168.2.82:3882
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2016-04-15 01:11:58,259 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 3 at election address /192.168.2.83:3883
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2016-04-15 01:11:58,260 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 400
2016-04-15 01:11:58,663 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 2 at election address /192.168.2.82:3882
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2016-04-15 01:11:58,666 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 3 at election address /192.168.2.83:3883
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2016-04-15 01:11:58,667 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 800
2016-04-15 01:11:59,470 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 2 at election address /192.168.2.82:3882
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2016-04-15 01:11:59,472 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 3 at election address /192.168.2.83:3883
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2016-04-15 01:11:59,478 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 1600
2016-04-15 01:12:01,080 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 2 at election address /192.168.2.82:3882
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2016-04-15 01:12:01,084 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 3 at election address /192.168.2.83:3883
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2016-04-15 01:12:01,085 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 3200
2016-04-15 01:12:04,287 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 2 at election address /192.168.2.82:3882
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2016-04-15 01:12:04,290 [myid:1] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@382] - Cannot open channel to 3 at election address /192.168.2.83:3883
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:402)
at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:762)
2016-04-15 01:12:04,291 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@849] - Notification time out: 6400

启动edu-zk-02

这里三个节点中有两个节点启动，所以日志中不会显示错误信息

[root@edu-zk-02 root]# who
yxq      pts/0        2016-04-15 00:50 (192.168.2.61)
[root@edu-zk-02 root]# /home/yxq/zookeeper/node-02/bin/zkServer.sh start
JMX enabled by default
Using config: /home/yxq/zookeeper/node-02/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@edu-zk-02 root]# ls
anaconda-ks.cfg  Desktop  Documents  Downloads  install.log  install.log.syslog  Music  Pictures  Public  Templates  Videos  zookeeper.out

查看是否正常启动

其中，QuorumPeerMain 是 zookeeper 进程，说明启动正常

[root@edu-zk-02 root]# jps
2929 QuorumPeerMain
2986 Jps

[root@edu-zk-02 root]# tail -500f zookeeper.out 
2016-04-15 01:15:24,042 [myid:] - INFO  [main:QuorumPeerConfig@103] - Reading configuration from: /home/yxq/zookeeper/node-02/bin/../conf/zoo.cfg
2016-04-15 01:15:24,051 [myid:] - INFO  [main:QuorumPeerConfig@340] - Defaulting to majority quorums
2016-04-15 01:15:24,055 [myid:2] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2016-04-15 01:15:24,056 [myid:2] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2016-04-15 01:15:24,056 [myid:2] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2016-04-15 01:15:24,064 [myid:2] - INFO  [main:QuorumPeerMain@127] - Starting quorum peer
2016-04-15 01:15:24,078 [myid:2] - INFO  [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2182
2016-04-15 01:15:24,100 [myid:2] - INFO  [main:QuorumPeer@959] - tickTime set to 2000
2016-04-15 01:15:24,100 [myid:2] - INFO  [main:QuorumPeer@979] - minSessionTimeout set to -1
2016-04-15 01:15:24,100 [myid:2] - INFO  [main:QuorumPeer@990] - maxSessionTimeout set to -1
2016-04-15 01:15:24,100 [myid:2] - INFO  [main:QuorumPeer@1005] - initLimit set to 10
2016-04-15 01:15:24,118 [myid:2] - INFO  [Thread-1:QuorumCnxManager$Listener@504] - My election bind port: /192.168.2.82:3882
2016-04-15 01:15:24,126 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:QuorumPeer@714] - LOOKING
2016-04-15 01:15:24,127 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:FastLeaderElection@815] - New election. My id =  2, proposed zxid=0x0
2016-04-15 01:15:24,132 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state)
2016-04-15 01:15:24,134 [myid:2] - WARN  [WorkerSender[myid=2]:QuorumCnxManager@382] - Cannot open channel to 3 at election address /192.168.2.83:3883
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:744)
2016-04-15 01:15:24,136 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@597] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state)
2016-04-15 01:15:24,138 [myid:2] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@597] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x1 (n.peerEpoch) LOOKING (my state)
2016-04-15 01:15:24,340 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:QuorumPeer@796] - LEADING
2016-04-15 01:15:24,345 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Leader@60] - TCP NoDelay set to: true
2016-04-15 01:15:24,360 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-04-15 01:15:24,361 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:host.name=edu-zk-02
2016-04-15 01:15:24,361 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:java.version=1.7.0_45
2016-04-15 01:15:24,361 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:java.vendor=Oracle Corporation
2016-04-15 01:15:24,362 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:java.home=/usr/java/jdk1.7.0_45/jre
2016-04-15 01:15:24,362 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:java.class.path=/home/yxq/zookeeper/node-02/bin/../build/classes:/home/yxq/zookeeper/node-02/bin/../build/lib/*.jar:/home/yxq/zookeeper/node-02/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/yxq/zookeeper/node-02/bin/../lib/slf4j-api-1.6.1.jar:/home/yxq/zookeeper/node-02/bin/../lib/netty-3.7.0.Final.jar:/home/yxq/zookeeper/node-02/bin/../lib/log4j-1.2.16.jar:/home/yxq/zookeeper/node-02/bin/../lib/jline-0.9.94.jar:/home/yxq/zookeeper/node-02/bin/../zookeeper-3.4.6.jar:/home/yxq/zookeeper/node-02/bin/../src/java/lib/*.jar:/home/yxq/zookeeper/node-02/bin/../conf:/usr/jdk/lib/
2016-04-15 01:15:24,362 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-04-15 01:15:24,362 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:java.io.tmpdir=/tmp
2016-04-15 01:15:24,363 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:java.compiler=<NA>
2016-04-15 01:15:24,363 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:os.name=Linux
2016-04-15 01:15:24,363 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:os.arch=amd64
2016-04-15 01:15:24,363 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:os.version=2.6.32-573.18.1.el6.x86_64
2016-04-15 01:15:24,364 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:user.name=root
2016-04-15 01:15:24,364 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:user.home=/root
2016-04-15 01:15:24,364 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Environment@100] - Server environment:user.dir=/root
2016-04-15 01:15:24,367 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:ZooKeeperServer@162] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/yxq/zookeeper/node-02/logs/version-2 snapdir /home/yxq/zookeeper/node-02/data/version-2
2016-04-15 01:15:24,369 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Leader@358] - LEADING - LEADER ELECTION TOOK - 241
2016-04-15 01:15:24,380 [myid:2] - INFO  [LearnerHandler-/192.168.2.81:48238:LearnerHandler@330] - Follower sid: 1 : info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@43cfd989
2016-04-15 01:15:24,395 [myid:2] - INFO  [LearnerHandler-/192.168.2.81:48238:LearnerHandler@385] - Synchronizing with Follower sid: 1 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2016-04-15 01:15:24,396 [myid:2] - INFO  [LearnerHandler-/192.168.2.81:48238:LearnerHandler@462] - Sending DIFF
2016-04-15 01:15:24,403 [myid:2] - INFO  [LearnerHandler-/192.168.2.81:48238:LearnerHandler@522] - Received NEWLEADER-ACK message from 1
2016-04-15 01:15:24,410 [myid:2] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Leader@943] - Have quorum of supporters, sids: [ 1,2 ]; starting up and setting last processed zxid: 0x200000000

先不启动edu-zk-03,

这里我们使用zkServer.sh status查看主从节点

leader:主节点

[root@edu-zk-02 root]# /home/yxq/zookeeper/node-02/bin/zkServer.sh status
JMX enabled by default
Using config: /home/yxq/zookeeper/node-02/bin/../conf/zoo.cfg
Mode: leader
[root@edu-zk-02 root]# 

follower从节点

[root@edu-zk-01 root]# /home/yxq/zookeeper/node-01/bin/zkServer.sh status
JMX enabled by default
Using config: /home/yxq/zookeeper/node-01/bin/../conf/zoo.cfg
Mode: follower
[root@edu-zk-01 root]# 

11、配置 zookeeper 开机使用yxq用户启动：

[root@edu-zk-01 root]# vi /etc/rc.local 
[root@edu-zk-01 root]# cat /etc/rc.local | grep yxq
su - yxq -c '/home/yxq/zookeeper/node-01/bin/zkServer.sh start'
[root@edu-zk-01 root]# 

[root@edu-zk-02 root]# vi /etc/rc.local 
[root@edu-zk-02 root]# cat /etc/rc.local | grep yxq
su - yxq -c '/home/yxq/zookeeper/node-02/bin/zkServer.sh start'
[root@edu-zk-02 root]# 

[root@edu-zk-03 ~]# vi /etc/rc.local 
[root@edu-zk-03 ~]# cat /etc/rc.local | grep yxq
su - yxq -c '/home/yxq/zookeeper/node-03/bin/zkServer.sh start'
[root@edu-zk-03 ~]# 

二、安装 Dubbo 管控台（基础篇有讲,此处重点讲管控台如何链接集群）：
Dubbo 管控台可以对注册到 zookeeper 注册中心的服务或服务消费者进行管理，但管控台是否正常对Dubbo 服务没有影响，管控台也不需要高可用，因此可以单节点部署。
IP: 192.168.2.81
部署容器：Tomcat7
端口：8080

root@edu-zk-01 webapps]# cd /root
[root@edu-zk-01 root]# ls
anaconda-ks.cfg  Desktop  Documents  Downloads  dubbo-governance.log  install.log  install.log.syslog  Music  Pictures  Public  Templates  Videos  zookeeper.out
[root@edu-zk-01 root]# cd /home/yxq/zookeeper/tomcat7/webapps/ROOT/
[root@edu-zk-01 ROOT]# ls
crossdomain.xml  css  dubbo-governance.log  favicon.ico  images  js  META-INF  SpryAssets  WEB-INF
[root@edu-zk-01 ROOT]# cd WEB-INF/
[root@edu-zk-01 WEB-INF]# ls
classes  common  dubbo.properties  forms  i18n  lib  log4j.xml  templates  webx-governance.xml  webx-home.xml  web.xml  webx-personal.xml  webx-sysinfo.xml  webx-sysmanage.xml  webx.xml
[root@edu-zk-01 WEB-INF]# vi dubbo.properties 
[root@edu-zk-01 WEB-INF]# cat dubbo.properties | grep re
dubbo.registry.address=zookeeper://192.168.2.81:2181?backup=192.168.2.82:2182,192.168.2.83:2183

三、测试集群高可用

启用edu-zk-03,这里是从节点

[root@edu-zk-03 conf]# /home/yxq/zookeeper/node-03/bin/zkServer.sh start
JMX enabled by default
Using config: /home/yxq/zookeeper/node-03/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@edu-zk-03 conf]# /home/yxq/zookeeper/node-03/bin/zkServer.sh status
JMX enabled by default
Using config: /home/yxq/zookeeper/node-03/bin/../conf/zoo.cfg
Mode: follower

我们再关掉edu-zk-02的leader节点

[root@edu-zk-02 root]# /home/yxq/zookeeper/node-02/bin/zkServer.sh stop
JMX enabled by default
Using config: /home/yxq/zookeeper/node-02/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED

我们再查看edu-zk-03 status

[root@edu-zk-03 conf]# /home/yxq/zookeeper/node-03/bin/zkServer.sh status
JMX enabled by default
Using config: /home/yxq/zookeeper/node-03/bin/../conf/zoo.cfg
Mode: leader

这时edu-zk-03变成了leader,

provider控制台信息

016-04-15 16:53:15,516  WARN [ClientCnxn.java:1089] : Session 0x15418fd25210000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
2016-04-15 16:53:15,517 DEBUG [ClientCnxnSocketNIO.java:192] : Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:798)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:426)
at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:189)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
2016-04-15 16:53:15,518 DEBUG [ClientCnxnSocketNIO.java:199] : Ignoring exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:815)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:434)
at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:196)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
2016-04-15 16:53:16,506  INFO [ClientCnxn.java:966] : Opening socket connection to server 192.168.2.83/192.168.2.83:2183. Will not attempt to authenticate using SASL (unknown error)
2016-04-15 16:53:16,507  INFO [ClientCnxn.java:849] : Socket connection established to 192.168.2.83/192.168.2.83:2183, initiating session
2016-04-15 16:53:16,510 DEBUG [ClientCnxn.java:889] : Session establishment request sent on 192.168.2.83/192.168.2.83:2183
2016-04-15 16:53:16,515  INFO [ClientCnxn.java:1207] : Session establishment complete on server 192.168.2.83/192.168.2.83:2183, sessionid = 0x15418fd25210000, negotiated timeout = 30000
2016-04-15 16:53:16,515 DEBUG [ZkClient.java:351] : Received event: WatchedEvent state:SyncConnected type:None path:null
2016-04-15 16:53:16,516  INFO [ZkClient.java:449] : zookeeper state changed (SyncConnected)
2016-04-15 16:53:16,516 DEBUG [ZkEventThread.java:88] : New event: ZkEvent[State changed to SyncConnected sent to com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient$1@20f8f517]
2016-04-15 16:53:16,516 DEBUG [ZkClient.java:395] : Leaving process event
2016-04-15 16:53:16,516 DEBUG [ZkEventThread.java:69] : Delivering event #2 ZkEvent[State changed to SyncConnected sent to com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient$1@20f8f517]
2016-04-15 16:53:16,517 DEBUG [ZkEventThread.java:79] : Delivering event #2 done
JMX enabled by default
Using config: /home/yxq/zookeeper/node-03/bin/../conf/zoo.cfg
Mode: leader

我们再关闭edu-zk-01节点

这里provider控制台信息报错(集群节点没有过半),这时dubbo管理台是可以正常运行的，但是集群是无用，提供者服务是缓存的，如果重启dubbo管理控制台，会发现启动不了，但是如果再启动一个节点edu-zk-01或edu-zk-02,管理控制台又立即可以运行起来。

java.net.SocketException: Connection reset by peer: shutdown
at sun.nio.ch.Net.shutdown(Native Method)
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:819)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:434)
at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:196)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
2016-04-15 16:58:09,848  INFO [ClientCnxn.java:966] : Opening socket connection to server 192.168.2.81/192.168.2.81:2181. Will not attempt to authenticate using SASL (unknown error)
2016-04-15 16:58:10,848  WARN [ClientCnxn.java:1089] : Session 0x15418fd25210000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
2016-04-15 16:58:10,848 DEBUG [ClientCnxnSocketNIO.java:192] : Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:798)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:426)
at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:189)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
2016-04-15 16:58:10,849 DEBUG [ClientCnxnSocketNIO.java:199] : Ignoring exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:815)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:434)
at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:196)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
2016-04-15 16:58:11,552  INFO [ClientCnxn.java:966] : Opening socket connection to server 192.168.2.82/192.168.2.82:2182. Will not attempt to authenticate using SASL (unknown error)
2016-04-15 16:58:12,553  WARN [ClientCnxn.java:1089] : Session 0x15418fd25210000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
2016-04-15 16:58:12,554 DEBUG [ClientCnxnSocketNIO.java:192] : Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:798)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:426)
at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:189)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
2016-04-15 16:58:12,555 DEBUG [ClientCnxnSocketNIO.java:199] : Ignoring exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:815)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:434)
at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:196)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
2016-04-15 16:58:13,909  INFO [ClientCnxn.java:966] : Opening socket connection to server 192.168.2.83/192.168.2.83:2183. Will not attempt to authenticate using SASL (unknown error)
2016-04-15 16:58:13,910  INFO [ClientCnxn.java:849] : Socket connection established to 192.168.2.83/192.168.2.83:2183, initiating session
2016-04-15 16:58:13,911 DEBUG [ClientCnxn.java:889] : Session establishment request sent on 192.168.2.83/192.168.2.83:2183
2016-04-15 16:58:13,914  INFO [ClientCnxn.java:1085] : Unable to read additional data from server sessionid 0x15418fd25210000, likely server has closed socket, closing socket connection and attempting reconnect
2016-04-15 16:58:13,915 DEBUG [ClientCnxnSocketNIO.java:199] : Ignoring exception during shutdown output