豌豆夹Redis解决方案Codis安装使用
豌豆夹Redis解决方案Codis安装使用
1.安装
1.1 Golang环境
Golang的安装非常简单，因为官网被墙，可以从国内镜像如studygolang.com下载。
[root@vm root]$ tar -C /usr/local -zxf go1.4.2.linux-amd64.tar.gz
[root@vm root]$ vim /etc/profile
export GOROOT=/usr/local/go
export PATH=$GOROOT/bin:$PATH
export GOPATH=/home/user/go
[root@vm root]$ source /etc/profile
[root@vm root]$ go version注意：操作系统的软件仓库如apt-get或yum安装的Golang版本可能比较老，要卸载后重新手动安装。以我的Linux Mint为例，安装时apt-get install golang指定的其实是个meta，卸载时要指定apt-get remove golang-go才能卸载干净。
1.2 Codis源码包
GitHub上的官方安装指导是这样写的：
[root@vm root]$ go get -d github.com/wandoulabs/codis
[root@vm root]$ cd $GOPATH/src/github.com/wandoulabs/codis
[root@vm root]$ ./bootstrap.sh
[root@vm root]$ make gotest这种方式会从GitHub上master下载最新的代码，但实践发现有时master代码可能会有问题。所以保险起见，还是下载稳定的release包吧。
1.3 编译安装
下载stable包后，我这里用的是最新的2.0.0版本包。手动创建出go的下载目录，将源码包解压后放进去，这样就手动完成了通过go get下载源码一步，避免用master分支代码的不稳定性。
[root@vm root] mkdir -p /home/root/go/src/github.com/wandoulabs/codis
[root@vm root] tar -xzvf ...
[root@vm root] cd codis
[root@vm codis] find . -name "configure" | xargs chmod +x
[root@vm codis] find . -name "*.sh" | xargs chmod +x [root@vm codis] ./bootstrap.sh
CC hyperloglog.o
CC latency.o
CC sparkline.o
CC slots.o
CC redis-cli.o
CC redis-benchmark.o
CC redis-check-dump.o
CC redis-check-aof.o
LINK redis-benchmark
LINK redis-check-dump
LINK redis-check-aof
LINK redis-server
INSTALL redis-sentinel
LINK redis-cli
Hint: It's a good idea to run 'make test' ;)
make[2]: Leaving directory `/home/root/go/src/github.com/wandoulabs/codis/extern/redis-2.8.21/src'
make[1]: Leaving directory `/home/root/go/src/github.com/wandoulabs/codis/extern/redis-2.8.21'
ln -s codis-server-2.8.21 bin/codis-server
go test ./pkg/... ./cmd/... -race
? github.com/wandoulabs/codis/pkg/env [no test files]
ok github.com/wandoulabs/codis/pkg/models 5.057s
? github.com/wandoulabs/codis/pkg/proxy/group [no test files]
ok github.com/wandoulabs/codis/pkg/proxy/redis 3.061s
ok github.com/wandoulabs/codis/pkg/proxy/router 17.916s
? github.com/wandoulabs/codis/pkg/proxy/router/topology [no test files]
? github.com/wandoulabs/codis/pkg/utils [no test files]
? github.com/wandoulabs/codis/pkg/utils/assert [no test files]
? github.com/wandoulabs/codis/pkg/utils/atomic2 [no test files]
ok github.com/wandoulabs/codis/pkg/utils/bytesize 0.025s
? github.com/wandoulabs/codis/pkg/utils/errors [no test files]
? github.com/wandoulabs/codis/pkg/utils/log [no test files]
? github.com/wandoulabs/codis/pkg/utils/trace [no test files]
? github.com/wandoulabs/codis/cmd/cconfig [no test files]
? github.com/wandoulabs/codis/cmd/proxy [no test files]2.启动
之前参照官网上的“流程”一节，被“坑”苦了…鼓捣了老半天也不行，要么就是Dashboard端口不对了，要么就是Slot初始化不了了，都要放弃了。发现正确的步骤在codis/sample/usage.md里，作者都写好了启动脚本。
[root@vm sample] ./start_dashboard.sh
[root@vm sample] ./start_redis.sh
[root@vm sample] ./add_group.sh
[root@vm sample] ./initslot.sh
[root@vm sample] ./start_proxy.sh
[root@vm sample] ./set_proxy_online.sh首先启动Dashboard，然后启动四个Redis实例，为这四个Redis实例创建四个Group，初始化Slot，启动一个Proxy实例，将Proxy置为Online状态。其中像新建Group、置为Online这种操作其实在Dashboard上都能做，但作者造福大家都给准备好了。一气呵成啊！刚才还自己瞎鼓捣了半天…
3.观察ZooKeeper
Codis使用ZooKeeper做协调，下面就看一下ZooKeeper里存的到底都是些什么。
Codis会根据我们在config.ini中配置的product_name作为ZooKeeper中的路径。
[zk: localhost:2181(CONNECTED) 34] ls /zk/codis/db_test
[fence, slots, servers, proxy, migrate_tasks, dashboard, LOCK, actions, ActionResponse]3.1 Slot信息
前面initslots初始化的所有Slot都保存在slots路径下，每个Slot是一个结点。
[zk: localhost:2181(CONNECTED) 35] ls /zk/codis/db_test/slots
[slot_240, slot_241, slot_242, slot_244, slot_243, slot_246, slot_245, slot_248, slot_247, slot_249, slot_804, slot_803, slot_802, slot_801, slot_808, slot_807, slot_806, slot_805, slot_800, slot_230, slot_231,
... ... ...
slot_832, slot_286, slot_672, slot_675, slot_674, slot_670, slot_671, slot_4, slot_3, slot_2, slot_1, slot_8, slot_7, slot_6, slot_5, slot_9, slot_0, slot_600, slot_602, slot_601, slot_608, slot_607, slot_609, slot_604, slot_603, slot_606, slot_605, slot_205, slot_206, slot_203, slot_204, slot_201, slot_202, slot_200, slot_209, slot_207, slot_208]随便选取一个Slot结点，用get命令能够查看结点上附着的数据：
[zk: localhost:2181(CONNECTED) 8] get /zk/codis/db_test/slots/slot_208
{"product_name":"test","id":208,"group_id":5,"state":{"status":"online","migrate_status":{"from":-1,"to":-1},"last_op_ts":"0"}}3.2 Proxy信息
[zk: localhost:2181(CONNECTED) 40] ls /zk/codis/db_test/proxy
[proxy_1]
[zk: localhost:2181(CONNECTED) 12] get /zk/codis/db_test/proxy/proxy_1
{"id":"proxy_1","addr":"BC-VM-edce4ac67d304079868c0bb265337bd4:19000","last_event":"","last_event_ts":0,"state":"online","description":"","debug_var_addr":"BC-VM-edce4ac67d304079868c0bb265337bd4:11000","pid":2946,"start_at":"2015-07-03 16:13:53.277706979 +0800 CST"}3.3 Server信息
Server的信息保存的只是Redis实例的IP地址和端口，所以并没有以JSON的形式附着在路径结点上，而是直接作为路径的一部分保存了。
[zk: localhost:2181(CONNECTED) 43] ls /zk/codis/db_test/servers
[group_1, group_2, group_3, group_4]
[zk: localhost:2181(CONNECTED) 44] ls /zk/codis/db_test/servers/group_1
[localhost:6380]
[zk: localhost:2181(CONNECTED) 45] ls /zk/codis/db_test/servers/group_2
[localhost:6381]
[zk: localhost:2181(CONNECTED) 46] ls /zk/codis/db_test/servers/group_3
[localhost:6382]
[zk: localhost:2181(CONNECTED) 47] ls /zk/codis/db_test/servers/group_4
[localhost:6383]3.4 其他信息
ZooKeeper中还保存了很多其他信息，这里就不一一解释了。
[zk: localhost:2181(CONNECTED) 53] ls /zk/codis/db_test/fence
[BC-VM-edce4ac67d304079868c0bb265337bd4:19000]
[zk: localhost:2181(CONNECTED) 24] get /zk/codis/db_test/dashboard
{"addr": "localhost:18087", "pid": 2871}
[zk: localhost:2181(CONNECTED) 33] get /zk/codis/db_test/actions/0000003698
{"type":"slot_changed","desc":"","target":{"product_name":"test","id":941,"group_id":5,"state":{"status":"online","migrate_status":{"from":-1,"to":-1},"last_op_ts":"0"}},"ts":"1435914788","receivers":["{\"id\":\"proxy_1\",\"addr\":\"BC-VM-edce4ac67d304079868c0bb265337bd4:19000\",\"last_event\":\"\",\"last_event_ts\":0,\"state\":\"online\",\"description\":\"\",\"debug_var_addr\":\"BC-VM-edce4ac67d304079868c0bb265337bd4:11000\",\"pid\":2946,\"start_at\":\"2015-07-03 16:13:53.277706979 +0800 CST\"}"]}4.测试
4.1 简单测试
因为Codis兼容大部分Redis操作，所以这里用原版Redis客户端连接到Proxy，测试一下保存和查询。
[root@vm redis-2.8.19] src/redis-cli -p 19000
127.0.0.1:19000> set person1 cdai
OK
127.0.0.1:19000> set student2 hank
OK
127.0.0.1:19000> set teacher3 carter
OK
127.0.0.1:19000> set manager4 rachel
OK
127.0.0.1:19000> get person1
"cdai"
127.0.0.1:19000> get student2
"hank"
127.0.0.1:19000> get teacher3
"carter"
127.0.0.1:19000> get manager4
"rachel"4.2 自动Rebalance
新添加一个Redis实例6379，并在Dashboard中为其新建Group 5，将新实例添加进去。
[root@vm sample] cp redis_conf/6380.conf redis_conf/6379.conf
[root@vm redis_conf] diff redis_conf/6379.conf redis_conf/6380.conf 41c41 < pidfile /var/run/redis_6379.pid
--- > pidfile /var/run/redis.pid
45c45
< port 6379 ---
> port 6380
177c177
< dbfilename 6379.rdb ---
> dbfilename 6380.rdb
[root@vm sample] nohup ../bin/codis-server ./redis_conf/6379.conf &> ./log/redis_6379.log &重新balance之前先看一下目前四个key的分布情况：
- 6380保存manager4和student2
- 6381什么都没保存
- 6382保存teacher3
- 6383保存person1
[root@vm redis-2.8.19] src/redis-cli
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379> quit
[root@vm redis-2.8.19] src/redis-cli -p 6380
127.0.0.1:6380> keys *
1) "manager4"
2) "student2"
127.0.0.1:6380> quit
[root@vm redis-2.8.19] src/redis-cli -p 6381
127.0.0.1:6381> keys *
(empty list or set)
127.0.0.1:6381> quit
[root@vm redis-2.8.19] src/redis-cli -p 6382
127.0.0.1:6382> keys *
1) "teacher3"
127.0.0.1:6382> quit
[root@vm redis-2.8.19] src/redis-cli -p 6383
127.0.0.1:6383> keys *
1) "person1"
127.0.0.1:6383> quit现在点”Auto Balance”看刚才保存的四个key会发生什么。
在Migration过程中，ZooKeeper中的内容也有变化。/zk/codis/db_test/migrate_tasks保存了正在migrating的Slot：
[zk: localhost:2181(CONNECTED) 61] ls /zk/codis/db_test/migrate_tasks
[0000000079, 0000000078, 0000000082, 0000000083, 0000000080, 0000000081, 0000000086, 0000000087, 0000000084, 0000000085, 0000000118, 0000000119, 0000000116, 0000000117, 0000000114, 0000000115, 0000000112, 0000000113, 0000000110, 0000000111, 0000000120, 0000000200, 0000000201, 0000000202, 0000000107, 0000000203, 0000000108, 0000000109, 0000000103, 0000000104, 0000000105, 0000000106, 0000000100, 0000000101, 0000000102, 0000000152, 0000000153, 0000000150, 0000000151, 0000000146, 0000000145, 0000000144, 0000000143, 0000000149, 0000000148, 0000000147, 0000000160, 0000000161, 0000000162, 0000000163, 0000000164, 0000000155, 0000000154, 0000000157, 0000000156, 0000000159, 0000000158, 0000000130, 0000000131, 0000000128, 0000000127, 0000000126, 0000000125, 0000000124, 0000000123, 0000000122, 0000000121, 0000000129, 0000000140, 0000000141, 0000000142, 0000000137, 0000000136, 0000000139, 0000000138, 0000000133, 0000000132, 0000000135, 0000000134, 0000000193, 0000000192, 0000000191, 0000000190, 0000000197, 0000000196, 0000000195, 0000000194, 0000000189, 0000000187, 0000000188, 0000000198, 0000000199, 0000000175, 0000000174, 0000000173, 0000000172, 0000000171, 0000000170, 0000000169, 0000000099, 0000000167, 0000000168, 0000000165, 0000000166, 0000000098, 0000000184, 0000000097, 0000000183, 0000000096, 0000000186, 0000000095, 0000000185, 0000000094, 0000000180, 0000000093, 0000000092, 0000000182, 0000000091, 0000000181, 0000000090, 0000000176, 0000000177, 0000000088, 0000000178, 0000000089, 0000000179]迁移后的key分配是：
[root@vm redis-2.8.19] src/redis-cli
127.0.0.1:6379> keys *
1) "student2"
127.0.0.1:6379> quit
[root@vm redis-2.8.19] src/redis-cli -p 6380
127.0.0.1:6380> keys *
1) "manager4"
127.0.0.1:6380> quit
[root@vm redis-2.8.19] src/redis-cli -p 6381
127.0.0.1:6381> keys *
(empty list or set)
127.0.0.1:6381> quit
[root@vm redis-2.8.19] src/redis-cli -p 6382
127.0.0.1:6382> keys *
1) "teacher3"
127.0.0.1:6382> quit
[root@vm redis-2.8.19] src/redis-cli -p 6383
127.0.0.1:6383> keys *
1) "person1"
127.0.0.1:6383> quit