codis架构

codis架构图

Codis组件

Codis 3.x 由以下组件组成：

Codis FE：集群管理界面
- 多个集群实例共享可以共享同一个前端展示页面；
- 通过配置文件管理后端 codis-dashboard 列表，配置文件可自动更新。

Codis Dashboard：集群管理工具，支持 codis-proxy、codis-server 的添加、删除，以及据迁移等操作。在集群状态发生改变时，codis-dashboard 维护集群下所有 codis-proxy 的状态的一致性。
- 对于同一个业务集群而言，同一个时刻 codis-dashboard 只能有 0个或者1个；
- 所有对集群的修改都必须通过 codis-dashboard 完成。

redis-sentinel:redis类似的高可用机制支持自动主从切换（HA）
Storage: 外部存储（注册中心）例如FileSystem、ETCD、Zookeeper

Codis Proxy：客户端连接的 Redis 代理服务, 实现了 Redis 协议。除部分命令不支持以外，表现的和原生的 Redis 没有区别（就像 Twemproxy）。
- 对于同一个业务集群而言，可以同时部署多个 codis-proxy 实例；
- 不同 codis-proxy 之间由 codis-dashboard 保证状态同步。
Codis-Group: 没有对应的组件,而是一个虚拟的节点机制通过槽位映射机制实现proxy与server的映射

Codis Server(Codis Redis)：基于 redis-3.2.8 分支开发。增加了额外的数据结构，以支持 slot 有关的操作以及数据迁移指令。

Codis Admin：集群管理的命令行工具。
- 可用于控制 codis-proxy、codis-dashboard 状态以及访问外部存储。

Codis集群启动

Codis集群启动需要zookeeper或者etcd,我们以zookeeper为例，搭建Codis集群。

关于zookeeper的配置，可以参考zookeeper安装部署

我们使用3台主机搭建集群环境
配置如下：

hostName	ip
vc0	192.168.4.50
vc1	192.168.4.51
vc2	192.168.4.52

vc0-1分别配置环境变量，在~/.bashrc中增加

export CODIS_HOME=/storage/dep/go/src/github.com/CodisLabs/codis
export PATH=$PATH:$CODIS_HOME/bin

启动 Codis Dashboard

使用vc0作为主机,修改vc0的Codis Dashboard配置文件vim /storage/dep/go/src/github.com/CodisLabs/codis/config/dashboard.toml



##################################################
#                                                #
#                  Codis-Dashboard               #
#                                                #
##################################################

# Set Coordinator, only accept "zookeeper" & "etcd" & "filesystem".
# for zookeeper/etcd, coorinator_auth accept "user:password" 
# Quick Start
coordinator_name = "zookeeper"
coordinator_addr = "192.168.4.50:2181,192.168.4.51:2181,192.168.4.52:2181"
#coordinator_auth = ""

# Set Codis Product Name/Auth.
product_name = "codis-nasa"
product_auth = ""

# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:18080"

# Set arguments for data migration (only accept 'sync' & 'semi-async').
migration_method = "semi-async"
migration_parallel_slots = 100
migration_async_maxbulks = 200
migration_async_maxbytes = "32mb"
migration_async_numkeys = 500
migration_timeout = "30s"

# Set configs for redis sentinel.
sentinel_client_timeout = "10s"
sentinel_quorum = 2
sentinel_parallel_syncs = 1
sentinel_down_after = "30s"
sentinel_failover_timeout = "5m"
sentinel_notification_script = ""
sentinel_client_reconfig_script = ""

重点修改的参数有

coordinator_name
- 外部存储类型，接受 zookeeper/etcd,本例使用zookeeper

coordinator_addr
- 外部存储地址，使用zookeeper集群的客户端暴露访问地址，以逗号分开

product_name
- 区分不同的集群，注册到zookeeper也是用这个区分 ,根据自己需求起名

admin_addr
- 用于dashboard与codas-proxy,codis-server进行tcp通信

使用命令nohup codis-dashboard --ncpu=1 --config=/storage/dep/go/src/github.com/CodisLabs/codis/config/dashboard.toml --log=/storage/dep/go/src/github.com/CodisLabs/codis/log/dashboard.log --log-level=WARN & 启动

其中：
--ncpu=N 最大使用 CPU 个数
-c CONF, --config=CONF 指定启动配置文件
-l FILE, --log=FILE 设置 log 输出文件
--log-level=LEVEL 设置 log 输出等级：INFO,WARN,DEBUG,ERROR；默认INFO，推荐WARN

执行后查看日志 tail -100f /storage/dep/go/src/github.com/CodisLabs/codis/log/dashboard.log.2018-06-25,得到输出

2018/06/25 17:12:36 main.go:78: [WARN] set ncpu = 1
2018/06/25 17:12:36 topom.go:119: [WARN] create new topom:
{
    "token": "cf7a714f4756eec383928d11c2899196",
    "start_time": "2018-06-25 17:12:36.761816028 +0800 CST",
    "admin_addr": "vc0:18080",
    "product_name": "codis-nsa",
    "pid": 5364,
    "pwd": "/storage/dep/go/src/github.com/CodisLabs/codis/log",
    "sys": "Linux vc0 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux"
}
2018/06/25 17:12:36 main.go:142: [WARN] create topom with config
coordinator_name = "zookeeper"
coordinator_addr = "192.168.4.50:2181,192.168.4.51:2181,192.168.4.52:2181"
coordinator_auth = ""
admin_addr = "0.0.0.0:18080"
product_name = "codis-nsa"
product_auth = ""


......
......


2018/06/25 17:12:37 topom_sentinel.go:169: [WARN] rewatch sentinels = []
2018/06/25 17:12:37 main.go:179: [WARN] [0xc4202bed80] dashboard is working ...
2018/06/25 17:12:37 topom.go:429: [WARN] admin start service on [::]:18080

观察日志输出发现Codis Dashboard已经按照配置正常启动

启动 Codis PROXY

配置文件如下：

product_name = "codis-nasa" # 集群名称，同codis-dashboard中配置
product_auth = "" #集群密码

session_auth = ""

# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:11080"  # 用于和codis-dashboard进行tcp通信同步信息

# Set bind address for proxy, proto_type can be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".
proto_type = "tcp4" #端口协议类型
proxy_addr = "0.0.0.0:19000"  #用于redis客户端连接,支持redis协议，端口唯一

jodis_name = ""
jodis_addr = ""
jodis_auth = ""
jodis_timeout = "20s"
jodis_compatible = false

# Set datacenter of proxy.
proxy_datacenter = ""

# Set max number of alive sessions.
proxy_max_clients = 1000

# Set max offheap memory size. (0 to disable)
proxy_max_offheap_size = "1024mb"

# Set heap placeholder to reduce GC frequency.
proxy_heap_placeholder = "256mb"

# Proxy will ping backend redis (and clear 'MASTERDOWN' state) in a predefined interval. (0 to disable)
backend_ping_period = "5s"

# Set backend recv buffer size & timeout.
backend_recv_bufsize = "128kb"
backend_recv_timeout = "30s"

# Set backend send buffer & timeout.
backend_send_bufsize = "128kb"
backend_send_timeout = "30s"

# Set backend pipeline buffer size.
backend_max_pipeline = 20480

# Set backend never read replica groups, default is false
backend_primary_only = false

# Set backend parallel connections per server
backend_primary_parallel = 1
backend_replica_parallel = 1

# Set backend tcp keepalive period. (0 to disable)
backend_keepalive_period = "75s"

# Set number of databases of backend.
backend_number_databases = 16

# If there is no request from client for a long time, the connection will be closed. (0 to disable)
# Set session recv buffer size & timeout.
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"

# Set session send buffer size & timeout.
session_send_bufsize = "64kb"
session_send_timeout = "30s"

# Make sure this is higher than the max number of requests for each pipeline request, or your client may be blocked.
# Set session pipeline buffer size.
session_max_pipeline = 10000

# Set session tcp keepalive period. (0 to disable)
session_keepalive_period = "75s"

# Set session to be sensitive to failures. Default is false, instead of closing socket, proxy will send an error response to client.
session_break_on_failure = false

# Set metrics server (such as http://localhost:28000), proxy will report json formatted metrics to specified server in a predefined period.
metrics_report_server = ""
metrics_report_period = "1s"

# Set influxdb server (such as http://localhost:8086), proxy will report metrics to influxdb.
metrics_report_influxdb_server = ""
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = ""

# Set statsd server (such as localhost:8125), proxy will report metrics to statsd.
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""

使用命令nohup codis-proxy --ncpu=1 --config=/storage/dep/go/src/github.com/CodisLabs/codis/config/proxy.toml --log=/storage/dep/go/src/github.com/CodisLabs/codis/log/proxy.log --log-level=WARN & 启动

查看日志tail -f /storage/dep/go/src/github.com/CodisLabs/codis/log/proxy.log.2018-06-25

2018/06/25 18:18:40 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
2018/06/25 18:18:41 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
2018/06/25 18:18:43 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
2018/06/25 18:18:44 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...

可以观察到codis-proxy已经启动，但是游离的，没有加入codis-dashboard管理，也没有与任何codis-server建立联系,我们可以稍后通过codis-fe将proxy纳入管理

启动 Codis Server(Redis实例)

我们可以将vc0,vc1,vc2三个主机上分别启动一个redis,vc0使用端口6380,vc1使用端口6381,vc2使用端口6382,

对vc0-2分别添加配置文件

vc0中 /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6380.conf
vc1中 /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6381.conf
vc2中 /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6382.conf

vc0中主要修改port 6380和pidfile /tmp/redis_6380.pid以及logfile "/tmp/redis_6380.log"以及bind 127.0.0.1 192.168.4.50
vc1中主要修改port 6381和pidfile /tmp/redis_6381.pid以及logfile "/tmp/redis_6381.log"以及bind 127.0.0.1 192.168.4.51
vc0中主要修改port 6382和pidfile /tmp/redis_6382.pid以及logfile "/tmp/redis_6382.log"以及bind 127.0.0.1 192.168.4.52

在vc0-2中分别执行以下命令启动redis实例

codis-server /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6380.conf
codis-server /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6381.conf
codis-server /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6382.conf

启动 Codis FE(管理平台)

vc0中启动Codis FE nohup ./bin/codis-fe --ncpu=1 --log=/storage/dep/go/src/github.com/CodisLabs/codis/log/fe.log --log-level=WARN --zookeeper=127.0.0.1:2181 --listen=192.168.4.50:8080 &
注意上面的命令必须要在CODIS_HOME下执行

其中:

zookeeper=127.0.0.1:2181为zk的连接地址
listen=192.168.4.50:8080为管理界面的访问地址和端口

通过fe管理集群

通过fe增加Proxy

打开FE的控制页面在Proxy栏中增加之前配置好的Proxy，输入Proxy的连接地址192.168.4.50:11080

如图：

[图片上传失败...(image-74f13d-1530000375805)]

通过fe增加Group(redis组)

add-poxy-by-fe

codis安装部署