Memcached+Magent+Keepalived高可用架构部署

Memcached集群介绍

• 现实生产环境中，我们要实现memcached主从复制避免单点故障风险，往往需要部署高可用群集。这里我们使用magent代理memcached实现主从备份，从而来保证缓存数据的完整性。工作原理如下：

1）数据在到达memcached服务器前会经过magent，每次写数据到主memcached服务器，magent会同时写到从 服务器 上，并且主从服务器写的算法一样。

2）当主memcached服务器宕掉之后，magent会向从memcached服务器中读取数据

3）当memcached恢复后，magent将重新从主memcached中读取数据，此时由于主memcached刚恢复正常，所以在宕掉期间的数据是无法获取的，这也是magent的缺点

• 再通过keepalived设置VIP地址，提供给Memcached API客户端以及magent连接访问。

1）memcached主从复制这种架构，在程序连接时不知道应该连接哪个主服务器，所以需要用keepalived在前端加VIP地址实现高可用；

2）keepalived不断检测memcached主服务器的11211端口，如果检测到memcached服务发生宕机或者死机等情况，就会将VIP从主服务器移至从服务器，从而实现高可用。

Memcached高可用架构图：

实验环境介绍：                                            

服务器主机	IP地址	需要安装的软件包	操作系统
master	192.168.30.55 VIP:192.168.30.16	libevent-2.1.8-stable.tar.gz，memcached-1.5.6.tar.gz, magent-0.5.tar.gz，keepalived	CentOS7
backup	192.168.30.10 VIP:192.168.30.16	libevent-2.1.8-stable.tar.gz，memcached-1.5.6.tar.gz， keepalived	CentOS7
client	192.168.30.15	telnet	RedHat6.5

百度网盘免费提供软件包：

libevent-2.1.8-stable.tar.gz：https://pan.baidu.com/s/1TBVEyndXVHcMbxst-kpCuA

memcached-1.5.6.tar.gz：https://pan.baidu.com/s/1oO_mC6J3SV6PLVepjmi2eQ

magent-0.5.tar.gz：https://pan.baidu.com/s/1xoT3Px14RHu6QE-66nLtMQ

实验部署：

首先所有主机关闭防火墙，及selinux

systemctl stop firewalld.service（CentOS7上）或者service iptables stop（RedHat6.5上）
setenforce 0

配置memcached主缓存节点和从缓存节点（主，从基本步骤相同，个别不同步骤下面会单独说明）

• 将事先准备的软件包进行解压

tar xvf memcached-1.5.6.tar.gz -C /opt/

tar xvf libevent-2.1.8-stable.tar.gz -C /opt/  #事件通知库

• 安装编译环境包

yum install gcc gcc-c++ make -y

编译安装libvent

cd /opt/libevent-2.1.8-stable
./configure --prefix=/usr      #指定工作目录

make && make install

• 编译安装memcached

cd /opt/memcached-1.5.6

./configure \
--prefix=/usr/local/memcached \         #指定工作目录
--with-libevent=/usr     #指定libevent的安装路径

make && make install

ln -s /usr/local/memcached/bin/* /usr/local/bin/  #让系统识别命令

• 在主服务器安装magent，从服务器不用安装

mkdir /opt/magent

cd ～/
tar xvf magent-0.5.tar.gz -C /opt/magent/
ketama.c
magent.c
ketama.h
Makefile

cd /opt/magent/

编辑配置文件 

vim ketama.h  

#ifndef SSIZE_MAX        #将原来的参数修改为SSIZE_MAX
#define SSIZE_MAX 32767        #将原来的参数修改为SSIZE_MAX 32767

 

修改Makefile

vim Makefile

LIBS = -levent -lm     #在后面加上-lm

 

编译（完成后会生成一个magent的可执行文件）

make         

复制mgent命令到系统管理中，便于使用

cp magent /usr/bin/  #将magent可执行文件加入usr可执行命令中

• 把产生的magent文件远程复制到从服务器上

[root@localhost magent]# scp magent 192.168.30.10:/usr/bin
The authenticity of host '192.168.30.10 (192.168.30.10)' can't be established.
ECDSA key fingerprint is SHA256:/5yAWf89joXesnoqczcLtVetMK2DIseKpZOcWkC+7V4.
ECDSA key fingerprint is MD5:a1:a4:ef:ff:84:46:0b:0c:fb:1d:de:56:4e:36:89:15.
Are you sure you want to continue connecting (yes/no)? yes                             #输入yes，同意复制
Warning: Permanently added '192.168.30.10' (ECDSA) to the list of known hosts.
[email protected]'s password:                                         #输入从服务器的root密码
magent                           100%  112KB   4.9MB/s   00:00

• 开启memcached缓存数据库（主，从都开启）

memcached -d -m 32m -p 11211 -u root

• 检查memcached进程是否运行

netstat -antp | grep 11211
tcp        0      0 0.0.0.0:11211           0.0.0.0:*               LISTEN      57782/memcached    
tcp6       0      0 :::11211                :::*                    LISTEN      57782/memcached

接着在主从服务器上安装配置keepalived

yum install keepalived -y

• 修改配置文件keepalived.conf

vim /etc/keepalived/keepalived.conf
其余设置项删除掉，就留下面的设置

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id MAGENT_HA          #指定router_id， 从服务器为MAGENT_HB
}

vrrp_script magent {                            #定义函数
    script "/opt/shell/magent.sh"
    interval 2
}

vrrp_instance VI_1 {
    state MASTER                #从服务器为BACKUP
    interface ens33                   #本地网卡名称
    virtual_router_id 51        #从服务器id不同
    priority 100                #从服务器优先级要小于主的
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {             #调用函数
     magent
    }
    virtual_ipaddress {
        192.168.30.16              #设置VIP
    }

• 主服务器上的keepalived配置

[root@localhost magent]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
  router_id MAGENT_HA
}

vrrp_script magent {
    script "/opt/shell/magent.sh"
    interval 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }

   track_script {
        magent
    }
    virtual_ipaddress {
        192.168.30.16
    }
}

• 从服务器上的keepalived配置

[root@localhost memcached-1.5.6]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id MAGENT_HB
}

vrrp_script magent {
    script "/opt/shell/magent.sh"
    interval 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 52
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }

   track_script {
        magent
    }
    virtual_ipaddress {
        192.168.30.16
    }
}

在主服务器上建立/opt/shell目录，创建magent.sh脚本

mkdir /opt/shell
vim /opt/shell/magent.sh

#!/bin/bash
K=`ps -ef | grep keepalived | grep -v grep | wc -l`
if [ $K -gt 0 ]; then
        magent -u root -n 51200 -l 192.168.30.16 -p 12000 -s 192.168.30.55:11211 -b 192.168.30.10:11211
else
pkill -9 magent
fi

• 其中的参数说明

-n 51200 //定义用户最大连接数
-l 192.168.175.188 //指定虚拟IP
-p 12000  //指定端口号
-s //指定主缓存服务器
-b //指定从缓存服务器

• 赋予脚本执行权限

chmod +x magent.sh      

• 启动keepalived服务
  systemctl start keepalived.service

• 查看12000端口的进程（magent）是否运行

netstat -antp | grep 12000
tcp        0      0 192.168.30.16:12000     0.0.0.0:*               LISTEN      22180/magent

• 查看系统公共日志文件/var/log/messages ，检测keepalived功能是否启用

vim /var/log/messages

Sep 26 12:20:42 localhost Keepalived_vrrp[24260]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 26 12:20:43 localhost Keepalived_vrrp[24260]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 26 12:20:43 localhost Keepalived_vrrp[24260]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 26 12:20:43 localhost Keepalived_vrrp[24260]: Sending gratuitous ARP on ens33 for 192.168.30.16
Sep 26 12:20:43 localhost Keepalived_vrrp[24260]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.30.16

同样在从服务器上建立/opt/shell目录，创建magent.sh脚本

mkdir /opt/shell
vim /opt/shell/magent.sh

#!/bin/bash
K=`ip addr | grep 192.168.30.16 | grep -v grep | wc -l`
if [ $K -gt 0 ]; then
        magent -u root -n 51200 -l 192.168.30.16 -p 12000 -s 192.168.30.55:11211 -b 192.168.30.10:11211
else
pkill -9 magent
fi 

• 赋予脚本执行权限

chmod +x magent.sh      

• 启动keepalived服务
  systemctl start keepalived.service

• 查看12000端口的进程（magent）是否运行

netstat -antp | grep 12000
tcp        0      0 192.168.30.16:12000     0.0.0.0:*               LISTEN      22180/magent

• 查看系统公共日志文件/var/log/messages ，检测keepalived功能是否启用

vim /var/log/messages

Sep 26 12:32:50 localhost Keepalived_vrrp[22129]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 26 12:32:50 localhost Keepalived_vrrp[22129]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

Sep 26 12:32:51 localhost Keepalived_vrrp[22129]: VRRP_Script(magent) succeeded

在client上测试

• 安装telnet，

yum -y install telnet

• 连接VIP，端口12000进行远程登录memcached群集，测试主从复制同步，添加键值username

[root@localhost ~]# telnet 192.168.30.16 12000

Trying 192.168.30.16...

Connected to 192.168.30.16.

Escape character is '^]'.

add username 0 0 6

abcdef

STORED

quit

Connection closed by foreign host.

• 连接master主服务器，查看username，同步成功
  [root@localhost ~]# telnet 192.168.30.55 11211
  Trying 192.168.30.55...
  Connected to 192.168.30.55.
  Escape character is '^]'.
  get username
  VALUE username 0 6
  abcdef
  END
  quit
  Connection closed by foreign host.

• 连接backup从服务器，查看username，同步成功
  [root@localhost ~]# telnet 192.168.30.10 11211
  Trying 192.168.30.10...
  Connected to 192.168.30.10.
  Escape character is '^]'.
  get username
  VALUE username 0 6
  abcdef
  END
  quit
  Connection closed by foreign host.

测试故障转移

• 在主服务器端关闭memcached服务

• 确认从服务器端memcached服务正常运行

• 在client端通过VIP连接群集，检测群集仍然正常运行