codis的proxy層HA

對Java使用者來說，可以使用經過修改過的Jedis-------Jodis，來實作proxy層的HA。它會通過監控zk上的注冊資訊來實時獲得目前可用的proxy清單，既可以保證高可用性，也可以通過輪流請求所有的proxy實作負載均衡。

jodis的位址如下：

https://github.com/wandoulabs/codis/tree/master/extern/jodis

具體使用如下：

JedisResourcePool jedisPool = new RoundRobinJedisPool("zkserver:2181", 30000, "/zk/codis/db_xxx/proxy", new JedisPoolConfig());
try (Jedis jedis = jedisPool.getResource()) {
    jedis.set("foo", "bar");
    String value = jedis.get("foo");
}
           

其中部分jedis參數設定如下：

//可用連接配接執行個體的最大數目，預設值為8；
//如果指派為-1，則表示不限制；如果pool已經配置設定了maxActive個jedis執行個體，則此時pool的狀态為exhausted(耗盡)。
private static int MAX_ACTIVE = 10;
//控制一個pool最多有多少個狀态為idle(空閑的)的jedis執行個體，預設值也是8。
private static int MAX_IDLE = 5;
//等待可用連接配接的最大時間，機關毫秒，預設值為-1，表示永不逾時。如果超過等待時間，則直接抛出JedisConnectionException；
private static int MAX_WAIT = 3000;
private static int TIMEOUT = 5000;
           

在實際運作過程中會報錯，部分錯誤如下：

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool 

at redis.clients.util.Pool.getResource(Pool.java:50) 

at redis.clients.jedis.JedisPool.getResource(JedisPool.java:86) 

at xt.city.edi.dao.redis.RedisUtil.getJedis(RedisUtil.java:81) 

at xt.city.edi.service.account.pojo.DefaultAccountManager.addUserlist(DefaultAccountManager.java:172) 

……

……

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object 

at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:449) 

at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) 

at redis.clients.util.Pool.getResource(Pool.java:48) 

正如上面提示：

我們jedis設定的連接配接池設定的太小，當連接配接池全被占用後，在“MAX_WAIT”的等待時間内，沒有執行個體被釋放，進而逾時等待，導緻could not get a resource from the pool。

在zookeeper的輸出日志zookeeper.out也可以看到：

2015-07-24 10:15:35,247 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:[email protected]] - Accepted socket connection from /192.168.1.119:30275 

2015-07-24 10:15:35,254 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:[email protected]] - Client attempting to establish new session at /192.168.1.119:30275 

2015-07-24 10:15:35,257 [myid:1] - INFO [CommitProcessor:1:[email protected]] - Established session 0x14e67e71be5003d with negotiated timeout 5000 for client /192.168.1.119:30275 

2015-07-24 10:20:24,576 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:[email protected]] - caught end of stream exception 

EndOfStreamException: Unable to read additional data from client sessionid 0x14e67e71be5003d, likely client has closed socket 

at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) 

at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) 

at java.lang.Thread.run(Thread.java:745) 

2015-07-24 10:20:24,577 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:[email protected]] - Closed socket connection for client /192.168.1.119:30275 which had sessionid 0x14e67e71be5003d 

2015-07-24 10:20:30,000 [myid:1] - INFO [SessionTracker:[email protected]] - Expiring session 0x14e67e71be5003d, timeout of 5000ms exceeded 

2015-07-24 10:20:30,001 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::[email protected]] - Processed session termination for sessionid: 0x14e67e71be5003dG

從上面紅色标注處可以得出結論：

jodis用戶端逾時了會關閉socket，導緻zookeeper剛建立的session過期釋放。

解決方案：

重新設定以下參數，将連接配接池中的連接配接執行個體和空閑執行個體增大，另外适當增大逾時等待時間，以保證連接配接執行個體被釋放。

MAX_ACTIVE=1024

MAX_IDLE = 200

MAX_WAIT = 10000