轻量的Memcached代理Twemproxy的部署
轻量的Memcached代理Twemproxy的部署
Twemproxy(又称为nutcracker)是一个轻量级的Redis和Memcached代理，主要用来减少对后端缓存服务器的连接数。由Twitter开源出来的缓存服务器集群管理工具，主要用来弥补Redis和Memcached对集群(cluster)管理指出的不足。
Twemproxy是一个快速的单线程代理程序，支持Memcached ASCII协议和更新的Redis协议。
Twemproxy最了不起的地方就在于它能在节点失败的时候卸载它，然后可以在一段时间以后重新尝试（随即）连接，又或者可以严格按照配置文件中写的键与服务器之间对应关系进行连接。
安装部署
现有测试机：192.168.11.51/52/68
先在51和52测试机上安装好libevent和memcached，分别启动两个memcached实例；
然后在68上安装好twemproxy，配置好参数，启动twemproxy实例。
安装和启动memcached实例
详细步骤，请参见之前的博文《Memcached 1.4.22安装和配置》，分别启动如下实例：
/usr/local/bin/memcached -d -m 128 -u memcached -l 192.168.11.51 -p 11211 -c 1024 -P /var/run/memcached/memcached1.pid /usr/local/bin/memcached -d -m 128 -u memcached -l 192.168.11.51 -p 11212 -c 1024 -P /var/run/memcached/memcached2.pid /usr/local/bin/memcached -d -m 128 -u memcached -l 192.168.11.52 -p 11211 -c 1024 -P /var/run/memcached/memcached1.pid /usr/local/bin/memcached -d -m 128 -u memcached -l 192.168.11.52 -p 11212 -c 1024 -P /var/run/memcached/memcached2.pid
安装和启动twemproxy实例
1、安装autoconf
cd /tmp wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz tar zxvf autoconf-2.69.tar.gz cd autoconf-2.69 ./configure --prefix=/usr/ make && make install
2、安装twemproxy
cd /tmp wget https://github.com/twitter/twemproxy/archive/master.zip unzip master.zip -d /usr/local/ cd /usr/local mv twemproxy-master twemproxy cd twemproxy CFLAGS="-ggdb3 -O0" autoreconf -fvi ./configure --prefix=/usr/local/twemproxy --enable-debug=log make && make install
3、查看帮助
[root@test01 twemproxy]# ./sbin/nutcracker -h
This is nutcracker-0.4.0 Usage: nutcracker [-?hVdDt] [-v verbosity level] [-o output file] [-c conf file] [-s stats port] [-a stats addr] [-i stats interval] [-p pid file] [-m mbuf size] Options: -h, --help : this help -V, --version : show version and exit -t, --test-conf : test configuration for syntax errors and exit -d, --daemonize : run as a daemon -D, --describe-stats : print stats description and exit -v, --verbose=N : set logging level (default: 5, min: 0, max: 11) -o, --output=S : set logging file (default: stderr) -c, --conf-file=S : set configuration file (default: conf/nutcracker.yml) -s, --stats-port=N : set stats monitoring port (default: 22222) -a, --stats-addr=S : set stats monitoring ip (default: 0.0.0.0) -i, --stats-interval=N : set stats aggregation interval in msec (default: 30000 msec) -p, --pid-file=S : set pid file (default: off) -m, --mbuf-size=N : set size of mbuf chunk in bytes (default: 16384 bytes)
4、修改配置文件
mkdir /etc/nutcracker cp ./conf/nutcracker.yml /etc/nutcracker/ vi /etc/nutcracker/nutcracker.yml
memcached: listen: 192.168.11.55:22121 hash: fnvla_64 distribution: ketama timeout: 400 backlog: 1024 preconnect: true auto_eject_hosts: true server_retry_timeout: 30000 server_failure_limit: 3 servers: - 192.168.11.51:11211:1 - 192.168.11.51:11212:1 - 192.168.11.52:11211:1 - 192.168.11.52:11212:1
参数解析：
listen: 启动twemproxy服务的IP和端口
hash: 指定具体的哈希函数
distribution: 指定具体的哈希算法
preconnect: 一个布尔值，如果该控件的nutcracker前端连接在这个池上的所有服务器进程启动。默认值为假
auto_eject_hosts: 是否在结点无法响应的时候临时摘除结点
server_retry_timeout: 重试的时间（毫秒）
server_failure_limit: 结点故障多少次就算摘除掉
servers: 下面表示所有的memcached节点（IP:端口号:权重）
5、配置以服务启动
cp ./scripts/nutcracker.init /etc/init.d/nutcracker chmod 755 /etc/init.d/nutcracker vi /etc/init.d/nutcracker
1. 新增定义daemo
daemon="/usr/local/twemproxy/sbin/nutcracker"
2. 替换所有
daemon --user ${USER} ${prog} $OPTIONS
为
${daemo} $OPTIONS
chkconfig --add nutcracker chkconfig --level 35 nutcracker on chkconfig --list nutcracker
vi /etc/profile
在里面加入：
export PATH="$PATH:/usr/local/twemproxy/sbin"
. /etc/profile echo $PATH
6、测试配置并启动服务
nutcracker -t -c /etc/nutcracker/nutcracker.yml service nutcracker start
数据写入测试
[root@test01 init.d]# telnet 192.168.11.55 11211
Trying 192.168.11.55... Connected to 192.168.11.55. Escape character is '^]'. set key1 0 0 1 1 STORED set key2 0 0 2 22 STORED set key3 0 0 3 333 STORED set key4 0 0 4 4444 STORED set key5 0 0 5 55555 STORED quit Connection closed by foreign host.
[root@test01 ~]# telnet 192.168.11.51 11211
Trying 192.168.11.51... Connected to 192.168.11.51. Escape character is '^]'. get key1 END get key2 END get key3 END get key4 END get key5 END quit Connection closed by foreign host.
[root@test01 ~]# telnet 192.168.11.51 11212
Trying 192.168.11.51... Connected to 192.168.11.51. Escape character is '^]'. get key1 END get key2 END get key3 END get key4 END get key5 END quit Connection closed by foreign host.
[root@test02 ~]# telnet 192.168.11.52 11211
Trying 192.168.11.52... Connected to 192.168.11.52. Escape character is '^]'. get key1 END get key2 END get key3 END get key4 END get key5 END quit Connection closed by foreign host.
[root@test02 ~]# telnet 192.168.11.52 11212
Trying 192.168.11.52... Connected to 192.168.11.52. Escape character is '^]'. get key1 VALUE key1 0 1 1 END get key2 VALUE key2 0 2 22 END get key3 VALUE key3 0 3 333 END get key4 VALUE key4 0 4 4444 END get key5 VALUE key5 0 5 55555 END quit Connection closed by foreign host.
可以看到数据全部从52的11212端口获取到。
现在停掉52的11212端口服务。
继续往代理写数据。
[root@test03 init.d]# telnet 192.168.11.55 11211
Trying 192.168.11.55... Connected to 192.168.11.55. Escape character is '^]'. set username 0 0 6 ryanxu STORED set aa 0 0 2 aa STORED quit Connection closed by foreign host.
[root@test03 init.d]# telnet 192.168.11.55 11211
Trying 192.168.11.55... Connected to 192.168.11.55. Escape character is '^]'. get key1 SERVER_ERROR Connection refused Connection closed by foreign host.
[root@test03 init.d]# telnet 192.168.11.55 11211
Trying 192.168.11.55... Connected to 192.168.11.55. Escape character is '^]'. get key1 SERVER_ERROR Connection refused Connection closed by foreign host.
[root@test03 init.d]# telnet 192.168.11.55 11211
Trying 192.168.11.55... Connected to 192.168.11.55. Escape character is '^]'. get key1 END get key2 END get key3 END quit Connection closed by foreign host.
一台memcached 挂掉后，twemproxy 能够自动摘除。恢复后，twemproxy 能够自动识别、恢复并重新加入到 memcached 组中重新使用。
问题总结
1）.yml配置文件中每个参数值对分隔符”:”后需要有一个空格。
2）不同层次的参数需要缩进区分，最好使用tab键缩进，否则nutcracker进程不能启动。
3）在auto_eject_hosts: true的时候，关闭一个memcached实例后，写入数据还是提示“(error) ERR Connection refused”。这个与server_retry_timeout参数设置太小有关，30000是一个很好的选择。