基于nutcracker 的redis分布式缓存的实现的注意
1 TwemProxy配置经验
What’s awesome about Twemproxy is that it can be configured both to disable nodes on failure, and retry after some time, or to stick to the specified keys -> servers map. This means that it is suitable both for sharding a Redis data set when Redis is used as a data store (disabling the node ejection), and when Redis is using as a cache, enabling node-ejection for cheap (as in simple, not as in bad quality) high availability.
The bottom line here is: if you enable node-ejection your data may end into other nodes when a node fails, so there is no guarantee about consistency. On the other side if you disable node-ejection you need to have a per-instance high availability setup, for example using automatic failover via Redis Sentinel.
当redis做存储的使用时为了保持数据的一致性，应该禁用auto_eject_hosts,也就是当某个节点失败之后并不删除该节点，所以，在每个redis实例上需要额外做一些高可用的配置，比如使用Redis Sentinel(开发中)。
当redis做缓存的使用的时候应该启用auto_eject_hosts，当某个节点失败的时候将该节点删除，虽然丧失了数据的一致性，但作为缓存使用，保证了这个集群的高可用性。
1）不支持mutiple keys的原因，将这个放在用户设计上，避免从mutiple keys上进行聚合等操作的时间
Currently is AFAIK even more strict than Redis Cluster that instead allows MULTI/EXEC blocks if all the commands are about the same key.
But IMHO it's the way to go, distribute the subset you can distribute efficiently, and pose this as a design challenge early to the user, instead to invest a big amount of resources into "just works" implementations that try to aggregate data from multiple instances, but that will hardly be fast enough once you start to have serious loads because of too big constant times to move data around.
在twemproxy中，一些在redis中如get的命令，如果报错，会一直占有这个链接
Twemproxy在errors时，可以监控到error，但是无法做到用其他node代替当前node进行操作，只是发出一个 A SLAVE OF NOONE commond
twemproxy is already able to monitor instance errors, count the number of errors, and eject the node when enough errors are detected. Well it is a shame it is not able to take slave nodes as alternatives, and instead of eject nodes use the alternate nodes just after sending a SLAVE OF NOONE command. This would turn it into an HA solution as well
在twemproxy中，对failover失效转化的处理，是用sentinel configuration有规律的更新服务器表
··2) Or alternatively, I would love if it could be able to work in tandem with Redis Sentinel, checking the Sentinel configuration regularly to upgrade the servers table if a failover happened.
· 还有一种替代的方法，是有一个hot-configure,在失败的时候，可是转化到这个配置文件，进行代理操作proxy ASAP
3) Another alternative is to provide a way to hot-configure twemproxy so that on fail overs Sentinel could switch the configuration of the proxy ASAP.
启动参数
Usage: nutcracker [-?hVdDt] [-v verbosity level] [-o output file]
[-c conf file] [-s stats port] [-a stats addr]
[-i stats interval] [-p pid file] [-m mbuf size]
Options:
-h, --help : this help
-V, --version : show version and exit
-t, --test-conf : 测试配置文件是否存在或者是否存在语法错误
-d, --daemonize : 守护进程启动
-D, --describe-stats : 打印状态
-v, --verbosity=N : 设置日志级别 (default: 5, min: 0, max: 11)
-o, --output=S : 设置日志文件位置 (default: stderr)
-c, --conf-file=S : 设置配置文件位置 (default: conf/nutcracker.yml)
-s, --stats-port=N : 设置状态监控端口 (default: 22222)
-a, --stats-addr=S : 设置状态监控IP (default: 0.0.0.0)
-i, --stats-interval=N : 设置状态监控聚合时间间隔 (default: 30000 msec)
-p, --pid-file=S : 设置pid文件 (default: off)
-m, --mbuf-size=N : 设置缓存块的大小 (default: 16384 bytes)
一般我会这样启动：
nutcracker -d -c /usr/local/nutcracker/nutcracker.yml -o /usr/local/nutcracker/nutcracker.log
支持的命令
string command: set,get,getset,mget,getrange,incr,decr,incrby,incrbyfloat,decrby,setbit,setex,setnx,setrange,strlen, psetex keys command: del,dump,exists,expire,expireat,persist,pexpire,pttl,restore hashes command: hdel,hexists,hget,hgetall,hincrby,hincrbyfloat,hkeys,hlen,hmget,hmset,hset,hsetnx,hvals list command: lindex,linsert,llen,lpop,lpush,lpushx,lrange,lrem,lset,limit,rpop,rpoplpush,rpush,rpushx set command: sadd,scard,sdiff,sdiffstore,sinter,sinterstore,sismember, smembers,smove,spop,srandmember,serm,sunion,sunionstore sorted sets command: zadd,zcard,zcount,zincrby,zincrby,zinterstore,zrange,zrangebyscore,zrank,zrem,zremrangebyrank,zrevrangebyscore,zrevrank,zscore,zunionstore scripting command: eval,evalsha
暂不支持的命令
keys command:
keys,migrate,move object,randomkey,rename,renamenx,sort
strings command:
bitop,mset,msetnx
list command:
blpop,brpop,brpoplpush
scripting command:
script exists,script flush,script kill,script load
pub/sub command:(全部不支持)
psubscribe,publish,punsubscribe,subscribe,unsubscribe
server command:(全部不支持)
bgrewriteaof,bgsave,client kill,client list,config get,config set, config resetstat,dbsize,flushdb,flushall,lastsave,monitor,save,shutdown,slaveof
connection command:(全部不支持)
auth,echo,ping,quit,select
transactions command:(全部不支持)
discard,exec,multi,unwatch,watch
状态监控
twemproxy自带监控端口，默认端口为222222，可以在启动时候使用-s 选项指定。监控状态如下
[root@slave nutcracker]# nc 127.0.0.1 22222
{"service":"nutcracker", "source":"slave", "version":"0.2.4", "uptime":103, "timestamp":1380011450,
"redis1": {"client_eof":0, "client_err":0, "client_connections":0, "server_ejects":0, "forward_error":0, "fragments":0,
"server1": {"server_eof":0, "server_err":0, "server_timedout":0,"server_connections":0, "requests":0, "request_bytes":0, "responses":0, "response_bytes":0, "in_queue":0, "in_queue_bytes":0, "out_queue":0, "out_queue_bytes":0},
"server2": {"server_eof":0, "server_err":0, "server_timedout":0, "server_connections":0, "requests":0, "request_bytes":0, "responses":0, "response_bytes":0, "in_queue":0, "in_queue_bytes":0, "out_queue":0, "out_queue_bytes":0},
"server3": {"server_eof":0, "server_err":0, "server_timedout":0, "server_connections":0, "requests":0, "request_bytes":0, "responses":0, "response_bytes":0, "in_queue":0, "in_queue_bytes":0, "out_queue":0, "out_queue_bytes":0},
"server4": {"server_eof":0, "server_err":0, "server_timedout":0, "server_connections":0, "requests":0, "request_bytes":0, "responses":0, "response_bytes":0, "in_queue":0, "in_queue_bytes":0, "out_queue":0, "out_queue_bytes":0}}}
可以看出，返回的信息主要分为三部分，twemproxy 概要信息，每个集群信息，每个集群下每个实例的信息。 可以使用 nutcracker -D 或者 -describe-stats 打印所有的状态描述信息
$ nutcracker --describe-stats
pool stats:
client_eof "# eof on client connections"
client_err "# errors on client connections"
client_connections "# active client connections"
server_ejects "# times backend server was ejected"
forward_error "# times we encountered a forwarding error"
fragments "# fragments created from a multi-vector request"
server stats:
server_eof "# eof on server connections"
server_err "# errors on server connections"
server_timedout "# timeouts on server connections"
server_connections "# active server connections"
requests "# requests"
request_bytes "total request bytes"
responses "# responses"
response_bytes "total response bytes"
in_queue "# requests in incoming queue"
in_queue_bytes "current request bytes in incoming queue"
out_queue "# requests in outgoing queue"
out_queue_bytes "current request bytes in outgoing queue"
高可用
不存在单点的问题：开启多个相同配置的twemproxy实例，保持客户端链接到一个可用的twemproxy上。每个twemproxy上可以配置多个代理集群。
性能
根据 Redis 作者的测试结果，在大多数情况下，Twemproxy 的性能相当不错，直接操作 Redis 相比，最多只有20%的性能损失。这对于它带来的好处来说真的是微不足道了。唯一可能还有待改进的是其 MGET 操作的效率，其性能只有直接操作 Redis 的 50%。
redis-benchmark against redis-server
$redis-benchmark -h -q -t set,get,incr,lpush,lpop,sadd,spop,lpush,lrange -c 100 -p 6379
SET: 89285.71 requests per second
GET: 92592.59 requests per second
INCR: 89285.71 requests per second
LPUSH: 90090.09 requests per second
LPOP: 90090.09 requests per second
SADD: 90090.09 requests per second
SPOP: 93457.95 requests per second
LPUSH (needed to benchmark LRANGE): 89285.71 requests per second
LRANGE_100 (first 100 elements): 36496.35 requests per second
LRANGE_300 (first 300 elements): 15748.03 requests per second
LRANGE_500 (first 450 elements): 11135.86 requests per second
LRANGE_600 (first 600 elements): 8650.52 requests per second
redis-benchmark against nutcracker proxing redis-server
$ redis-benchmark -h -q -t set,get,incr,lpush,lpop,sadd,spop,lpush,lrange -c 100 -p 22121
SET: 85470.09 requests per second
GET: 86956.52 requests per second
INCR: 85470.09 requests per second
LPUSH: 84745.77 requests per second
LPOP: 86206.90 requests per second
SADD: 84745.77 requests per second
SPOP: 86956.52 requests per second
LPUSH (needed to benchmark LRANGE): 84745.77 requests per second
LRANGE_100 (first 100 elements): 29761.90 requests per second
LRANGE_300 (first 300 elements): 12376.24 requests per second
LRANGE_500 (first 450 elements): 8605.85 requests per second
LRANGE_600 (first 600 elements): 6587.62 requests per second
问题和不足
- 不支持针对多个值的操作，比如取sets的子交并补等（MGET 和 DEL 除外）
- 不支持Redis的事务操作
- 出错提示还不够完善
启动参数
Usage: nutcracker [-?hVdDt] [-v verbosity level] [-o output file]
[-c conf file] [-s stats port] [-a stats addr]
[-i stats interval] [-p pid file] [-m mbuf size]
Options:
-h, --help : this help
-V, --version : show version and exit
-t, --test-conf : 测试配置文件是否存在或者是否存在语法错误
-d, --daemonize : 守护进程启动
-D, --describe-stats : 打印状态
-v, --verbosity=N : 设置日志级别 (default: 5, min: 0, max: 11)
-o, --output=S : 设置日志文件位置 (default: stderr)
-c, --conf-file=S : 设置配置文件位置 (default: conf/nutcracker.yml)
-s, --stats-port=N : 设置状态监控端口 (default: 22222)
-a, --stats-addr=S : 设置状态监控IP (default: 0.0.0.0)
-i, --stats-interval=N : 设置状态监控聚合时间间隔 (default: 30000 msec)
-p, --pid-file=S : 设置pid文件 (default: off)
-m, --mbuf-size=N : 设置缓存块的大小 (default: 16384 bytes)
一般我会这样启动：
nutcracker -d -c /usr/local/nutcracker/nutcracker.yml -o /usr/local/nutcracker/nutcracker.log
支持的命令
string command:
set,get,getset,mget,getrange,incr,decr,incrby,incrbyfloat,decrby,setbit,setex,setnx,setrange,strlen, psetex
keys command:
del,dump,exists,expire,expireat,persist,pexpire,pttl,restore
hashes command:
hdel,hexists,hget,hgetall,hincrby,hincrbyfloat,hkeys,hlen,hmget,hmset,hset,hsetnx,hvals
list command:
lindex,linsert,llen,lpop,lpush,lpushx,lrange,lrem,lset,limit,rpop,rpoplpush,rpush,rpushx
set command:
sadd,scard,sdiff,sdiffstore,sinter,sinterstore,sismember,
smembers,smove,spop,srandmember,serm,sunion,sunionstore
sorted sets command:
zadd,zcard,zcount,zincrby,zincrby,zinterstore,zrange,zrangebyscore,zrank,zrem,zremrangebyrank,zrevrangebyscore,zrevrank,zscore,zunionstore
scripting command:
eval,evalsha
暂不支持的命令
keys command:
keys,migrate,move object,randomkey,rename,renamenx,sort
strings command:
bitop,mset,msetnx
list command:
blpop,brpop,brpoplpush
scripting command:
script exists,script flush,script kill,script load
pub/sub command:(全部不支持)
psubscribe,publish,punsubscribe,subscribe,unsubscribe
server command:(全部不支持)
bgrewriteaof,bgsave,client kill,client list,config get,config set, config resetstat,dbsize,flushdb,flushall,lastsave,monitor,save,shutdown,slaveof
connection command:(全部不支持)
auth,echo,ping,quit,select
transactions command:(全部不支持)
discard,exec,multi,unwatch,watch
状态监控
twemproxy自带监控端口，默认端口为222222，可以在启动时候使用-s 选项指定。监控状态如下
[root@slave nutcracker]# nc 127.0.0.1 22222
{"service":"nutcracker", "source":"slave", "version":"0.2.4", "uptime":103, "timestamp":1380011450,
"redis1": {"client_eof":0, "client_err":0, "client_connections":0, "server_ejects":0, "forward_error":0, "fragments":0,
"server1": {"server_eof":0, "server_err":0, "server_timedout":0,"server_connections":0, "requests":0, "request_bytes":0, "responses":0, "response_bytes":0, "in_queue":0, "in_queue_bytes":0, "out_queue":0, "out_queue_bytes":0},
"server2": {"server_eof":0, "server_err":0, "server_timedout":0, "server_connections":0, "requests":0, "request_bytes":0, "responses":0, "response_bytes":0, "in_queue":0, "in_queue_bytes":0, "out_queue":0, "out_queue_bytes":0},
"server3": {"server_eof":0, "server_err":0, "server_timedout":0, "server_connections":0, "requests":0, "request_bytes":0, "responses":0, "response_bytes":0, "in_queue":0, "in_queue_bytes":0, "out_queue":0, "out_queue_bytes":0},
"server4": {"server_eof":0, "server_err":0, "server_timedout":0, "server_connections":0, "requests":0, "request_bytes":0, "responses":0, "response_bytes":0, "in_queue":0, "in_queue_bytes":0, "out_queue":0, "out_queue_bytes":0}}}
可以看出，返回的信息主要分为三部分，twemproxy 概要信息，每个集群信息，每个集群下每个实例的信息。 可以使用 nutcracker -D 或者 -describe-stats 打印所有的状态描述信息
$ nutcracker --describe-stats
pool stats:
client_eof "# eof on client connections"
client_err "# errors on client connections"
client_connections "# active client connections"
server_ejects "# times backend server was ejected"
forward_error "# times we encountered a forwarding error"
fragments "# fragments created from a multi-vector request"
server stats:
server_eof "# eof on server connections"
server_err "# errors on server connections"
server_timedout "# timeouts on server connections"
server_connections "# active server connections"
requests "# requests"
request_bytes "total request bytes"
responses "# responses"
response_bytes "total response bytes"
in_queue "# requests in incoming queue"
in_queue_bytes "current request bytes in incoming queue"
out_queue "# requests in outgoing queue"
out_queue_bytes "current request bytes in outgoing queue"
高可用
不存在单点的问题：开启多个相同配置的twemproxy实例，保持客户端链接到一个可用的twemproxy上。每个twemproxy上可以配置多个代理集群。
性能
根据 Redis 作者的测试结果，在大多数情况下，Twemproxy 的性能相当不错，直接操作 Redis 相比，最多只有20%的性能损失。这对于它带来的好处来说真的是微不足道了。唯一可能还有待改进的是其 MGET 操作的效率，其性能只有直接操作 Redis 的 50%。
redis-benchmark against redis-server
$redis-benchmark -h -q -t set,get,incr,lpush,lpop,sadd,spop,lpush,lrange -c 100 -p 6379
SET: 89285.71 requests per second
GET: 92592.59 requests per second
INCR: 89285.71 requests per second
LPUSH: 90090.09 requests per second
LPOP: 90090.09 requests per second
SADD: 90090.09 requests per second
SPOP: 93457.95 requests per second
LPUSH (needed to benchmark LRANGE): 89285.71 requests per second
LRANGE_100 (first 100 elements): 36496.35 requests per second
LRANGE_300 (first 300 elements): 15748.03 requests per second
LRANGE_500 (first 450 elements): 11135.86 requests per second
LRANGE_600 (first 600 elements): 8650.52 requests per second
redis-benchmark against nutcracker proxing redis-server
$ redis-benchmark -h -q -t set,get,incr,lpush,lpop,sadd,spop,lpush,lrange -c 100 -p 22121
SET: 85470.09 requests per second
GET: 86956.52 requests per second
INCR: 85470.09 requests per second
LPUSH: 84745.77 requests per second
LPOP: 86206.90 requests per second
SADD: 84745.77 requests per second
SPOP: 86956.52 requests per second
LPUSH (needed to benchmark LRANGE): 84745.77 requests per second
LRANGE_100 (first 100 elements): 29761.90 requests per second
LRANGE_300 (first 300 elements): 12376.24 requests per second
LRANGE_500 (first 450 elements): 8605.85 requests per second
LRANGE_600 (first 600 elements): 6587.62 requests per second
问题和不足
- 不支持针对多个值的操作，比如取sets的子交并补等（MGET 和 DEL 除外）
- 不支持Redis的事务操作
- 出错提示还不够完善