codis的proxy层HA

对Java用户来说，可以使用经过修改过的Jedis-------Jodis，来实现proxy层的HA。它会通过监控zk上的注册信息来实时获得当前可用的proxy列表，既可以保证高可用性，也可以通过轮流请求所有的proxy实现负载均衡。

jodis的地址如下：

https://github.com/wandoulabs/codis/tree/master/extern/jodis

具体使用如下：

JedisResourcePool jedisPool = new RoundRobinJedisPool("zkserver:2181", 30000, "/zk/codis/db_xxx/proxy", new JedisPoolConfig());
try (Jedis jedis = jedisPool.getResource()) {
    jedis.set("foo", "bar");
    String value = jedis.get("foo");
}

其中部分jedis参数设置如下：

//可用连接实例的最大数目，默认值为8；
//如果赋值为-1，则表示不限制；如果pool已经分配了maxActive个jedis实例，则此时pool的状态为exhausted(耗尽)。
private static int MAX_ACTIVE = 10;
//控制一个pool最多有多少个状态为idle(空闲的)的jedis实例，默认值也是8。
private static int MAX_IDLE = 5;
//等待可用连接的最大时间，单位毫秒，默认值为-1，表示永不超时。如果超过等待时间，则直接抛出JedisConnectionException；
private static int MAX_WAIT = 3000;
private static int TIMEOUT = 5000;

在实际运行过程中会报错，部分错误如下：

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool 
at redis.clients.util.Pool.getResource(Pool.java:50) 
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:86) 
at xt.city.edi.dao.redis.RedisUtil.getJedis(RedisUtil.java:81) 
at xt.city.edi.service.account.pojo.DefaultAccountManager.addUserlist(DefaultAccountManager.java:172) 
……
……
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object 
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449) 
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) 
at redis.clients.util.Pool.getResource(Pool.java:48) 

正如上面提示：

我们jedis设置的连接池设置的太小，当连接池全被占用后，在“MAX_WAIT”的等待时间内，没有实例被释放，从而超时等待，导致could not get a resource from the pool。

在zookeeper的输出日志zookeeper.out也可以看到：

2015-07-24 10:15:35,247 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /192.168.1.119:30275 
2015-07-24 10:15:35,254 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /192.168.1.119:30275 
2015-07-24 10:15:35,257 [myid:1] - INFO [CommitProcessor:1:ZooKeeperServer@617] - Established session 0x14e67e71be5003d with negotiated timeout 5000 for client /192.168.1.119:30275 
2015-07-24 10:20:24,576 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception 
EndOfStreamException: Unable to read additional data from client sessionid 0x14e67e71be5003d, likely client has closed socket 
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) 
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) 
at java.lang.Thread.run(Thread.java:745) 
2015-07-24 10:20:24,577 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /192.168.1.119:30275 which had sessionid 0x14e67e71be5003d 
2015-07-24 10:20:30,000 [myid:1] - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x14e67e71be5003d, timeout of 5000ms exceeded 
2015-07-24 10:20:30,001 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x14e67e71be5003dG

从上面红色标注处可以得出结论：

jodis客户端超时了会关闭socket，导致zookeeper刚建立的session过期释放。

解决方案：

重新设置以下参数，将连接池中的连接实例和空闲实例增大，另外适当增大超时等待时间，以保证连接实例被释放。

MAX_ACTIVE=1024

MAX_IDLE = 200

MAX_WAIT = 10000