Codis服务搭建成功案例

Codis服务搭建
一、 基础配置
服务器3台:zookeeper-1, zookeeper-2，zookeeper-3
操作系统：linux centos 7
配置hosts：
vi /etc/hosts
11.1.1.10 zookeeper-1
11.1.1.11 zookeeper-2
11.1.1.12 zookeeper-2
二、 软件安装
1、安装java环境
yum -y install java-1.7.0-openjdk-devel
java -version
java version "1.7.0_75"
密码：jodis不支持
2、安装zookeeper（3台服务都需要配置）
wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
tar zxvf zookeeper-3.4.6.tar.gz
mv zookeeper-3.4.6 /usr/local/zookeeper
mkdir -p /data/zookeeper/{data,logs}

配置zoo.cfg
vim /usr/local/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=5
syncLimit=2
dataDir=/data/zookeeper/data

dataLogDir=/data/zookeeper/logs

clientPort=2181
server.1=zookeeper-1:2888:3888
server.2=zookeeper-2:2888:3888
server.3=zookeeper-2:2888:3888
上述配置内容说明，可以参考
http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html#sc_RunningReplicatedZooKeeper

设置myid
在我们配置的dataDir指定的目录下面，创建一个myid文件，里面内容为一个数字，用来标识当前主机，conf/zoo.cfg文件中配置的server.X中X为什么数字，则myid文件中就输入这个数字
[root@zookeeper-1 ~]# echo 1 > /data/zookeeper/data/myid
[root@zookeeper-2 ~]# echo 2 > /data/zookeeper/data/myid
[root@zookeeper-3 ~]# echo 3 > /data/zookeeper/data/myid
设置环境变量
vim /etc/profile
export ZOOKEEPERPATH=/usr/local/zookeeper
GO_BIN=/usr/local/go/bin
export GOROOT=/usr/local/go
export CODISPATH=/usr/local/codis
export GOPATH=/data/gopath
export GO15VENDOREXPERIMENT=1
export PATH=$PATH:$GOROOT/bin:$ZOOKEEPERPATH/bin:$CODISPATH/bin:$GOPATH/bin

source /etc/profile
测试zoopkeeper
启动顺序zookeeper-1>zookeeper-2>zookeeper-3
[root@zookeeper-1 zookeeper]# zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@zookeeper-1 zookeeper]# zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader
可以看到最先开始启动的是leader，其他两个是follower

注意关闭防火墙：systemctl stop firewalld.service

3、安装go语言环境
(参考：https://github.com/CodisLabs/codis/blob/master/doc/tutorial_zh.md)
cd /home/deploy/
wget http://golangtc.com/static/go/ go1.5.2.linux-amd64.tar.gz

tar -zxvf go1.5.2.linux-amd64.tar.gz
4、安装codis
安装git
[root@vm-82-39-buy01-hp ~]# yum install -y git

配置环境变量
[root@vm-82-39-buy01-hp ~]# vim /etc/profile
export GOROOT=/usr/local/go

     export GOPATH=/home/deploy/gopath

     export CODISPATH=/usr/local/codis
     export PATH=/bin:$PATH:$GOROOT/bin:$GOPATH/bin:$CODISPATH/bin

[root@vm-82-39-buy01-hp ~]# source /etc/profile
[root@vm-82-39-buy01-hp ~]# go get -u -d github.com/CodisLabs/codis #这个需要几分钟下载共30M文件
cd $GOPATH/src/github.com/CodisLabs/codis
make
make gotest
没有错误后

mkdir -p /home/deploy/codis/data/codis_server/{logs,conf,data}
mkdir -p /usr/local/codis/{logs,conf,scripts}
cp ./config.ini / usr/local/codis/conf/
cp -rf bin /usr/local/codis/
5、配置运行codis
这是一个灰常麻烦的过程，要有心里准备
建立codis配置文件
vi usr/local/codis/conf/config.ini(zookeeper-1配置)
coordinator=zookeeper
zk=10.154.82.106:2181,10.154.82.107:2181,10.154.82.108:2181
product=codis-proxy
dashboard_addr=10.154.82.39:18087
password=
backend_ping_period=5
session_max_timeout=1800
session_max_bufsize=131072
session_max_pipeline=1024
zk_session_timeout=30000
proxy_id=proxy_1
vi usr/local/codis/conf /config.ini(zookeeper-2配置)
coordinator=zookeeper
zk=10.154.82.106:2181,10.154.82.107:2181,10.154.82.108:2181
product= codis-proxy
dashboard_addr=10.154.82.107:2181
password=
backend_ping_period=5
session_max_timeout=1800
session_max_bufsize=131072
session_max_pipeline=1024
zk_session_timeout=30000
proxy_id=proxy_2
vi usr/local/codis/conf /config.ini(zookeeper-3配置)
coordinator=zookeeper
zk=10.154.82.106:2181,10.154.82.107:2181,10.154.82.108:2181
product= codis-proxy
dashboard_addr=10.154.82.108:2181
password=
backend_ping_period=5
session_max_timeout=1800
session_max_bufsize=131072
session_max_pipeline=1024
zk_session_timeout=30000
proxy_id=proxy_3
建立codis-server服务配置文件
cd usr/local/codis/conf/
vi redis.conf
daemonize yes
pidfile /var/run/redis_6379.pid
port 6379
tcp-backlog 511
timeout 300
tcp-keepalive 0
loglevel notice
logfile "/data/codis_server/logs/redis_6379.log"
databases 16
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename 6379.rdb
dir /data/codis_server/data
slave-serve-stale-data yes
repl-disable-tcp-nodelay no
slave-priority 100
maxclients 10000
maxmemory 3gb
maxmemory-policy allkeys-lru
appendonly yes
appendfilename "6379_appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

将redis.conf配置文件 到其他codis集群机器上

6、启动顺序(三台服务器都需要启动)
A、启动zk
zkServer.sh start
B、启动codis
a、 启动dashboard
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini dashboard &
b、初始化slots
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini slots init
c、启动codis redis
/usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf
/usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf
/usr/local/codis/bin/codis-server /data/codis_server/conf/6389.conf
/usr/local/codis/bin/codis-server /data/codis_server/conf/6390.conf
d、添加 Redis Server Group
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 0 zookeeper-1:6379 master &
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 1 zookeeper-1:6380 master &
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 2 zookeeper-1:6389 master &
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 3 zookeeper-1:6390 master &

/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 0 zookeeper-2:6379 slave &
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 1 zookeeper-2:6380 slave &
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 2 zookeeper-2:6389 slave &
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 3 zookeeper-2:6390 slave &

e、分配slot
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/ config.ini slot range-set 0 341 1 online
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini slot range-set 342 682 2 online
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini slot range-set 683 1023 3 online

f、启动 codis-proxy
/usr/local/codis/bin/codis-proxy -c /usr/local/codis/conf/config.ini -L /data/log/codis-proxy.log --cpu=1 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000 &
g、上线proxy
/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini proxy online proxy_1
h、测试
http://192.168.1.221:18087/admin/

redis-cli -h 192.168.11.231 -p 19000
set pwd 123456
get pwd
set pwd2 123456
get pwd2

redis-cli -h 192.168.11.232 -p 19000
get pwd
get pwd2

redis-benchmark -h 192.168.1.221 -p 19000 -q -d 100  
SET/GET 100 bytes 检测host为192.168.1.221 端口为19000的redis服务器性能


redis-benchmark -h 192.168.1.222 -p 19000 -c 500 -n 10000 
500个并发连接，10000个请求，检测host为192.168.1.222 端口为19000的redis服务器性能

============================ok codis搭建完成=======================
参考文档：https://github.com/CodisLabs/codis