codis服务实例搭建

Codis服务搭建

一、基础配置

服务器2台:zookeeper-1, zookeeper-2

操作系统：linux centos 7

配置hosts：

    vi /etc/hosts

     11.1.1.10   zookeeper-1

        11.1.1.11   zookeeper-2

 

二、软件安装

1、安装java环境

yum -y install java-1.7.0-openjdk-devel

java -version
java version "1.7.0_75"

2、安装zookeeper（2台服务都需要配置）

wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
tar zxvf zookeeper-3.4.6.tar.gz
mv zookeeper-3.4.6 /usr/local/zookeeper
mkdir -p /data/zookeeper/{data,logs}

配置zoo.cfg
vim /usr/local/zookeeper/conf/zoo.cfg 
tickTime=2000
initLimit=5
syncLimit=2
dataDir=/data/zookeeper/data
#dataLogDir=/data/zookeeper/logs
clientPort=2181
server.1=zookeeper-1:2888:3888
server.2=zookeeper-2:2888:3888
上述配置内容说明，可以参考
http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html#sc_RunningReplicatedZooKeeper

设置myid
在我们配置的dataDir指定的目录下面，创建一个myid文件，里面内容为一个数字，用来标识当前主机，conf/zoo.cfg文件中配置的server.X中X为什么数字，则myid文件中就输入这个数字
[root@zookeeper-1 ~]# echo 1 > /data/zookeeper/data/myid

[root@zookeeper-2 ~]# echo 2 > /data/zookeeper/data/myid

设置环境变量

vim /etc/profile
export ZOOKEEPERPATH=/usr/local/zookeeper

GO_BIN=/usr/local/go/bin

export GOROOT=/usr/local/go

export CODISPATH=/usr/local/codis

export GOPATH=/data/gopath

export GO15VENDOREXPERIMENT=1

export PATH=$PATH:$GOROOT/bin:$ZOOKEEPERPATH/bin:$CODISPATH/bin:$GOPATH/bin


source /etc/profile

测试zoopkeeper

启动顺序zookeeper-1>zookeeper-2
[root@zookeeper-1 zookeeper]# zkServer.sh start  
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@zookeeper-1 zookeeper]# zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader
可以看到最先开始启动的是leader，其他两个是follower

注意关闭防火墙：systemctl stop firewalld.service

3、安装go语言环境

(参考：https://github.com/CodisLabs/codis/blob/master/doc/tutorial_zh.md)

cd /usr/local/

wget http://golangtc.com/static/go/go1.4.2.linux-amd64.tar.gz

注意：不要使用高于1.4.2的go环境包,codis暂时不支持

tar -zxvf go1.3.3.linux-amd64.tar.gz    

4、安装codis

yum install -y git
go get -u -d github.com/CodisLabs/codis #这个需要几分钟下载共30M文件

cd$GOPATH/src/github.com/CodisLabs/codis

make

make gotest

没有错误后

cp –rf  *  /usr/local/codis/

cd  /usr/local/codis/

 

5、配置运行codis

这是一个灰常麻烦的过程，要有心里准备

mkdir   /usr/local/codis/conf

建立codis配置文件

vi  conf/config_dashboard.ini(zookeeper-1配置)

coordinator=zookeeper

zk=zookeeper-1:2181,zookeeper-2:2181

product=test

dashboard_addr=zookeeper-1:18087

password=

backend_ping_period=5

session_max_timeout=1800

session_max_bufsize=131072

session_max_pipeline=1024

zk_session_timeout=30000

proxy_id=proxy_1

vi  conf/config_dashboard.ini(zookeeper-2配置)

coordinator=zookeeper

zk=zookeeper-1:2181,zookeeper-2:2181

product=test

dashboard_addr=zookeeper-1:18087

password=

backend_ping_period=5

session_max_timeout=1800

session_max_bufsize=131072

session_max_pipeline=1024

zk_session_timeout=30000

proxy_id=proxy_2

                     建立codis-server服务配置文件

                cd /data/codis_server/conf/

                vi       6379.conf

daemonize yes
pidfile /var/run/redis_6379.pid
port 6379
tcp-backlog 511
timeout 300
tcp-keepalive 0
loglevel notice
logfile "/data/codis_server/logs/redis_6379.log"
databases 16
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename 6379.rdb
dir /data/codis_server/data
slave-serve-stale-data yes
repl-disable-tcp-nodelay no
slave-priority 100
maxclients 10000
maxmemory 3gb
maxmemory-policy allkeys-lru
appendonly yes
appendfilename "6379_appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

 

复制6380、6389、6390配置文件
cp 6379.conf 6380.conf
cp 6379.conf 6389.conf
cp 6379.conf 6390.conf
sed -i 's/6379/6380/g' 6380.conf
sed -i 's/6379/6389/g' 6389.conf
sed -i 's/6379/6390/g' 6390.conf

 

 

 

6、修改系统内核参数

添加内核参数
echo "vm.overcommit_memory = 1" >>  /etc/sysctl.conf
sysctl -p
内核参数说明如下：  

                                                                                                overcommit_memory文件指定了内核针对内存分配的策略


，其值可以是0、1、2。               0，表示内核将检查是否有足够的可用内存供应用进程使用；如果有足够的可
用内存，内存申请允许；否则，内存申请失败，并把错误返回给应用进程。 
1，表示内核允许分配所有的物理内存，而不管当前的内存状态如何。
2，表示内核允许分配超过所有物理内存和交换空间总和的内存

7、启动顺序(两台服务器都需要启动)

            A、启动zk

                            zkServer.sh start

            B、启动codis

                     a、 启动dashboard

                            /usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini dashboard &

                     b、初始化slots

                            /usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini slots init

                     c、启动codis redis

                            /usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf

                            /usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf

/usr/local/codis/bin/codis-server /data/codis_server/conf/6389.conf

/usr/local/codis/bin/codis-server /data/codis_server/conf/6390.conf

                     d、添加 Redis Server Group

                            /usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 0 zookeeper-1:6379 master &

                            /usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 1 zookeeper-1:6380 master &

/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 2 zookeeper-1:6389 master &

/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 3 zookeeper-1:6390 master &

 

/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 0 zookeeper-2:6379 slave &

                            /usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 1 zookeeper-2:6380 slave &

/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 2 zookeeper-2:6389 slave &

/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini server add 3 zookeeper-2:6390 slave &

 

e、分配slot

/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/ config.ini slot range-set 0  341 1 online

/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini slot range-set 342 682 2 online

/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini slot range-set 683 1023 3 online

 

f、启动 codis-proxy

/usr/local/codis/bin/codis-proxy  -c /usr/local/codis/conf/config.ini -L /data/log/codis-proxy.log  --cpu=1 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000 &

g、上线proxy

/usr/local/codis/bin/codis-config -c /usr/local/codis/conf/config.ini proxy online proxy_1

h、测试

http://192.168.1.221:18087/admin/

 

redis-cli -h 192.168.11.231 -p 19000

         set pwd 123456

         get pwd

         set pwd2 123456    

         get pwd2

 

         redis-cli -h 192.168.11.232 -p 19000

         get pwd

         get pwd2

 

         redis-benchmark -h 192.168.1.221 -p 19000 -q -d 100 

         SET/GET 100 bytes 检测host为192.168.1.221 端口为19000的redis服务器性能

 

 

         redis-benchmark -h 192.168.1.222 -p 19000 -c 500 -n 10000

         500个并发连接，10000个请求，检测host为192.168.1.222 端口为19000的redis服务器性能

 

============================ok codis搭建完成=======================

 

参考文档：https://github.com/CodisLabs/codis