MHA+Lvs+Keepalived实现MySQL的高可用及读负载均衡_4（Lvs+Keepalived）

MHA+Lvs+Keepalived实现MySQL的高可用及读负载均衡_4（Lvs+Keepalived）

实现Masterha-manager高可用及MySQL从的读负载均衡的Lvs+Keepalived安装及配置

--20150211

今天发现一个问题——之前的方案解决masterha_manager的高可用，会致使两个Lvs服务器都down，经检查，发现是因为两台lvs上的/etc/app1.cnf不一致导致的，解决方法：

第一步：先在两台lvs服务器间创建ssh无密码连接

第二步：在Lvs-15.23的notify_master脚本的第一行添加“scp -P 1777 [email protected]:/etc/app1.cnf /etc/”，在Lvs-15.24的notify_master脚本的第一行添加“scp -P 1777 [email protected]:/etc/app1.cnf /etc/”  ，以实现两边的app1.cnf保持一致

在上述解决方案的基础上，假想这样一个场景。Master1 down了，MHA实现了new master的切换。你在完成对Master1的修复之后，将其change master到new master，此时只需更新当前Lvs主的app1.cnf，假使当前Lvs主是Lvs-15.23，那么在Lvs-15.23的app1.cnf里新增修复后的server Master1，然后在Lvs-15.23上执行masterha_stop --conf=/etc/app1.cnf，Lvs-15.24会将Lvs-15.23的app1.cnf同步到本地，并启动masterha_manager，实现masterha_manager的高可用。

 

一、安装ipvsadm

1 下载

http://www.linuxvirtualserver.org/software/ipvs.html

下载时需要检查服务器linux内核版本，以下载相对应版本的ipvsadm

 

2 安装

2.1 创建内核目录的链接文件

[[email protected] ~ 13:08:34]#ls /usr/src/kernels

2.6.18-308.el5-x86_64

[[email protected] ~ 13:08:42]#

ln -s /usr/src/kernels/2.6.18-308.el5-x86_64  /usr/src/linux

[[email protected] ~ 13:09:48]#ll /usr/src/linux

lrwxrwxrwx 1 root root 38 01-04 17:54 /usr/src/linux -> 

/usr/src/kernels/2.6.18-308.el5-x86_64

 

2.2 安装

[[email protected] xxm 13:13:00]#tar xf ipvsadm-1.24.tar.gz

[[email protected] xxm 13:13:11]#cd ipvsadm-1.24

[[email protected] ipvsadm-1.24 13:45:10]#make && make install

 

2.3 开启路由转发功能

[[email protected] ~ 13:46:25]#/bin/sed -i 's/ip_forward = 0/ip_forward = 1/p' /etc/sysctl.conf

[[email protected] ~ 13:46:45]#sysctl -p

 

二、安装Keepalived

1 下载

http://www.keepalived.org/

2 安装

2.1 软件安装

[[email protected] xxm 13:49:52]#tar xf keepalived-1.2.12.tar.gz 

[[email protected] xxm 13:50:02]#cd keepalived-1.2.12

[[email protected] keepalived-1.2.12 13:50:03]# 

./configure --prefix=/usr/local/keepalived

[[email protected] keepalived-1.2.12 13:51:38]#make && make install

 

2.2 相关文件cp

此操作的原因->keepalived服务启动脚本默认指定了参数路径，所以为了能够正常启动keepalived服务，需要执行上述操作，当然也可以不cp目标文件，直接vi修改目标文件，将相应的参数指向为真实文件即可。

/bin/cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d

/bin/cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig

/bin/cp /usr/local/keepalived/sbin/keepalived /usr/sbin

/bin/mkdir /etc/keepalived

 

2.3 分离Keepalived日志文件

2.3.1 

[[email protected] keepalived 13:56:52]#vi /etc/sysconfig/keepalived

[[email protected] keepalived 13:57:54]#tail -2 /etc/sysconfig/keepalived 

KEEPALIVED_OPTIONS="-D -d -S 0"

 

2.3.2 修改syslog.conf定义keepalived的日志

[[email protected] ~ 13:59:52]#tail -2 /etc/syslog.conf 

# Save keepalived messages to keepalived.log

local0.*                                                /var/log/keepalived.log

 

2.3.2 重启syslog服务

[[email protected] ~ 14:00:03]#/etc/init.d/syslog restart

 

三、配置Keepalived.conf

3.1 keepalived.conf文件

[[email protected] app1 14:03:38]#cd /etc/keepalived/

[[email protected] keepalived 14:04:35]#vi keepalived.conf

[[email protected] keepalived 14:04:57]#cat keepalived.conf

! Configuration File for keepalived

 

! Globals configurations

 

!!!!!!!!!!!!===全局配置===!!!!!!!!!!!

global_defs  {  ! 全局配置

notification_email  {  ! 通知邮件的收件人列表 

[email protected]

 } 

notification_email_from [email protected] ! 发件人地址

smtp_server 127.0.0.1 ! 使用的邮箱服务器

smtp_connect_timeout 30 ! 连接smtp server的timeout

router_id masterha_manager_1

 } 

!!!!!!!!!!!!===全局配置===!!!!!!!!!!!

 

! VRRP configurations

 

!!!!!!!!!!!!===VRRP配置===!!!!!!!!!!!

vrrp_script check_masterha_manager  {  ! 声明一个自定义调用系统服务监控脚本，'check_masterha_manager'为自定义

       script "/usr/local/scripts/check_masterha_manager.sh" ! 服务健康检查脚本，如果目标服务down，那么停掉本地                                                                   ! keepalived，实现failover

       interval 3      ! 脚本被调用的频率,当前是每3秒调用一次

        }

 

vrrp_sync_group mha  {  ! 声明一个vrrp group，'mha'为自定义,group里定义了一系列有相同特性的instance

group  { 

mha_app1 ! 自定义instan_1 'mha_app1'

 } 

 } 

 

vrrp_instance mha_app1  {  ! 定义一个vrrp实例，'mha_app1'为自定义，具体实现group里定义的instance

state BACKUP ! 起初状态为bakcup

interface eth0 ! 虚拟ip绑定的目标网卡 

 

virtual_router_id 61 ! 虚拟路由ID标记，主要是个标记作用 整数型，range 0~100

priority 100 ! 优先级定义，值越大，优先级越高。整数型，range 0~255

advert_int 1 ! 检查间隔时间

 

authentication  {  ! 主从间实例的秘钥配对验证

auth_type PASS ! 简单的password验证

auth_pass 1111 ! 密码值

 } 

virtual_ipaddress  {  ! 进行failover的虚拟ip

! 192.168.15.96/24 label eth0:1  ! 也可以这样简单的写

192.168.15.96/24   ! 也可以这样简单的写

 } 

 

        track_script  {         ! 跟踪监控脚本，调用之前定义的vrrp script 'check_masterha_manager'

                check_masterha_manager

                 }

 

nopreempt !不抢占 

smtp_alter ! failover后通过global里的邮件设置发通知邮件

notify_master "/usr/local/scripts/notify_master.sh"

notify_backup "/usr/local/scripts/notify_stop.sh"

notify_stop "/usr/local/scripts/notify_stop.sh"

 } 

 

!!!!!!!!!!!!===VRRP配置===!!!!!!!!!!!

 

 

!!!!!!!!!!!!===LVS配置===!!!!!!!!!!!

virtual_server_group MRead { 

192.168.15.96 36677

} 

 

virtual_server 192.168.15.96 36677 { 

lvs_sched rr ! LVS的调度算法

lvs_method DR ! LVS的模式

protocol TCP 

nat_mask 255.255.255.0

 

sorry_server 192.168.15.11 36677 ! 备机，可做故障迁移用

real_server 192.168.15.12 36677 { 

weight 1

inhibit_on_failure

TCP_CHECK {  

connect_port 36677

connect_timeout 10

nb_get_retry 3

                delay_before_retry 1

} 

}

real_server 192.168.15.13 36677 { 

weight 1

inhibit_on_failure

TCP_CHECK {  

connect_port 36677

connect_timeout 10

nb_get_retry 3

                delay_before_retry 1

} 

}

real_server 192.168.15.14 35577 { 

weight 1

inhibit_on_failure

TCP_CHECK {  

connect_port 36677

connect_timeout 10

nb_get_retry 3

                delay_before_retry 1

} 

}

} 

!!!!!!!!!!!!===LVS配置===!!!!!!!!!!!

 

Lvs-15.24的keepalived配置文件有以下3处与Lvs-15.23不同，其他均一样：

[[email protected] keepalived 15:14:08]#cat keepalived.conf_real

......

notification_email_from [email protected] ! 发件人地址

.......

router_id masterha_manager_2

......

virtual_router_id 61 ! 虚拟路由ID标记，主要是个标记作用 整数型，range 0~100

priority 50 ! 优先级定义，值越大，优先级越高。整数型，range 0~255

......

 

3.2 check_masterha_manager脚本

#!/bin/bash
###变量及函数定义###
MHA_OK=0 ##mha正常运行，MHA_OK为0，反之为1
CHECK_TIME=3 ##此次脚本检查共检查3次
VIP=192.168.4.96


function mha_check() {
/sbin/ip addr|grep ${VIP} > /dev/null 2>&1
if [ $? -eq 0  ]
then
/usr/bin/masterha_check_status --conf=/etc/app1.cnf > /dev/null 2>&1 #检查masterha_manager的运行状态
if [ $? -eq 2 ] 
then
MHA_OK=1
else
MHA_OK=0
fi
else
exit 0
fi
}
###变量及函数定义###


###实现三次检查###
while [ ${CHECK_TIME} -ne 0 ]
do
CHECK_TIME=$((${CHECK_TIME}-1))
mha_check

if [ ${MHA_OK} -eq 0 ]
then
CHECK_TIME=0
exit 0
fi


if  [ ${MHA_OK} -eq 1 ]  && [ ${CHECK_TIME} -eq 0 ]
then
/etc/init.d/keepalived stop
        exit 1
fi

sleep 1
done
###实现三次检查###

 

3.3 notify_master.sh脚本

[[email protected] ~ 14:14:27]#vi /usr/local/scripts/notify_master.sh

[[email protected] ~ 14:15:44]#cat /usr/local/scripts/notify_master.sh

#!/bin/bash

scp [email protected]:/etc/app1.cnf /etc/    #Lvs-15.24上要scp15.23的到本地

/usr/bin/nohup /usr/bin/masterha_manager --conf=/etc/app1.cnf --wait_on_monitor_error=10 --wait_on_failover_error=10 --remove_dead_master_conf < /dev/null > /var/log/masterha/app1/app1.log 2>&1 &

#/usr/bin/nohup /usr/bin/masterha_manager --conf=/etc/app1.cnf --wait_on_failover_error=10 --remove_dead_master_conf < /dev/null > /var/log/masterha/app1/app1.log 2>&1 &

 

3.4 notify_stop.sh脚本

[[email protected] ~ 14:15:47]#vi /usr/local/scripts/notify_stop.sh

[[email protected] ~ 14:16:41]#cat /usr/local/scripts/notify_stop.sh

#!/bin/bash

masterha_check_status --conf=/etc/app1.cnf|tr ' ' '\n'|grep pid > /dev/null 2>&1

if [ $? -eq 0  ]

then

kill -9 `masterha_check_status --conf=/etc/app1.cnf |cut -d' ' -f2|cut -d':' -f2|tr ')' ' '`

echo "kill command"

else

echo "no pid"

exit 0

fi

 

四、MySQL-slave上配置vip

在MySQL-15.11、MySQL-15.12、MySQL-15.13、MySQL-15.14上编写如下文件，启动并加入开机自启动

 

[[email protected] ~ 15:29:37]#vi /etc/init.d/lvsrs 

[[email protected] ~ 15:29:41]#ll /etc/init.d/lvsrs 

-rw-r--r-- 1 root root 1004 01-18 15:29 /etc/init.d/lvsrs

[[email protected] ~ 15:29:43]#chmod +x  /etc/init.d/lvsrs 

[[email protected] ~ 15:29:52]#cat  /etc/init.d/lvsrs 

#!/bin/bash

#chkconfig: 2345 80 90

#description:config VIP on LVS of Real Server

 

VIP=192.168.15.96

 

#./etc/rc.d/init.d/functions

case "$1" in

    start)

        echo " Start LVS  of  Real Server"

 /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up

 /sbin/route add -host ${VIP} dev lo:0

        echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

        echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

        echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

        echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce 

 /sbin/sysctl -p > /dev/null 2>&1

        ;;

    stop)

        /sbin/ifconfig lo:0 down

        echo "close LVS Director server"

        echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

        echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

        echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

        echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

 /sbin/sysctl -p > /dev/null 2>&1

        ;;

    *)

        echo "Usage: $0 {start|stop}"

        exit 1

esac

 

[[email protected] ~ 15:32:06]#chkconfig lvsrs on

[[email protected] ~ 15:32:30]#chkconfig --list lvsrs

lvsrs           0:关闭 1:关闭 2:启用 3:启用 4:启用 5:启用 6:关闭

 

 

 

五、测试

1 启动lvsrs

4台MySQL服务器启动lvsrs

注：MySQL-15.11也启动lvsrs，是因为将其定位了sorry_server，若没有此需求，当然可以不在MySQL-15.11上启动lvsrs。

 

以MySQL-15.12为例：

[[email protected] ~ 20:31:08]#/etc/init.d/lvsrs start

 Start LVS  of  Real Server

[[email protected] ~ 20:38:07]#ip addr|grep 15.96

    inet 192.168.15.96/32 brd 192.168.15.96 scope global lo:0

 

2 启动keepalived服务

两台Lvs服务器Lvs-15.23（主）、Lvs-15.24（从）启动keepalived服务

Lvs-15.23：

[[email protected] ~ 20:28:30]#/etc/init.d/keepalived start

启动 keepalived：                                          [确定]

You have new mail in /var/spool/mail/root

[[email protected] ~ 20:40:07]#ip addr|grep 15.96

    inet 192.168.15.96/24 scope global secondary eth0

[[email protected] ~ 20:40:15]#masterha_check_status --conf=/etc/app1.cnf 

app1 (pid:17521) is running(0:PING_OK), master:192.168.15.11

 

Lvs-15.24：

[[email protected] ~ 20:40:52]#/etc/init.d/keepalived start

启动 keepalived：                                          [确定]

[[email protected] ~ 20:40:57]#ip addr|grep 15.96

[[email protected] ~ 20:41:09]#masterha_check_status --conf=/etc/app1.cnf 

app1 is stopped(2:NOT_RUNNING).

 

3 测试masterha_manager及Lvs的failover

Lvs-15.23上的keepalived服务停止

[[email protected] ~ 20:40:27]#/etc/init.d/keepalived stop

停止 keepalived：                                          [确定]

[[email protected] ~ 20:42:43]#ip addr|grep 15.96

[[email protected] ~ 20:42:59]#masterha_check_status --conf=/etc/app1.cnf 

app1 is stopped(2:NOT_RUNNING).

 

查看Lvs-15.24的状态：

[[email protected] ~ 20:42:47]#ip addr|grep 15.96

    inet 192.168.15.96/24 scope global secondary eth0

[[email protected] ~ 20:43:29]#masterha_check_status --conf=/etc/app1.cnf 

app1 (pid:22419) is running(0:PING_OK), master:192.168.15.11

[[email protected] ~ 20:43:31]#ipvsadm -L

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  192.168.15.96:35577 rr

  -> MySQL-15.14:36677            Route   1      0          0         

  -> MySQL-15.13:36677            Route   1      0          0         

  -> MySQL-15.12:36677            Route   1      0          0        

 

 

用VIP访问试一下：

mysql -u'xxm_test' -p'123456' -h'192.168.15.96' -P'36677'

mysql -u'xxm_test' -p'123456' -h'192.168.15.95' -P'36677'

|||

均可正常访问