关于keepalive 问题处理

昨天新配置了Nginx Keepalived架构，今天上午来了VIP无法访问了，于是查看日志开始解决问题。
主服务器日志：
Sep 25 16:08:28 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 16:08:33 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:08:23 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:08:23 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:15:49 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:15:49 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:15:50 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1Sep 25 17:30:09 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new electionSep 25 17:30:51 modoor Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 1.1.1.1
从服务器日志：
Sep 25 15:58:19 wap Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 25 15:58:19 wap Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 1.1.1.1
Sep 25 15:58:24 wap Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 1.1.1.1
Sep 25 16:08:25 wap Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 25 16:11:32 wap Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 25 16:48:01 wap Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 25 16:48:02 wap Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 25 16:48:02 wap Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 25 16:48:02 wap Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 1.1.1.1
Sep 25 16:48:05 wap Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Sep 25 16:48:05 wap Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 25 16:48:05 wap Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 25 17:08:19 wap Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 25 17:08:19 wap Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Sep 25 17:08:19 wap Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 25 17:08:23 wap Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 25 17:08:24 wap Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 25 17:08:24 wap Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 25 17:08:24 wap Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 1.1.1.1
Sep 25 17:08:25 wap Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
从日志分析：主备在反复的切换。于是查看自己的配置文件
我的配置文件是这样的：
global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_script chk_http_port {
                script "/opt/check_nginx.sh"
                interval 2
                weight 2
}
vrrp_instance VI_1 {
    state BACKUP        
    interface eth0
    virtual_router_id 50    
    mcast_src_ip 1.1.1.100
    priority 100                  
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
track_script {
        chk_http_port ### 执行监控的服务
        }
    virtual_ipaddress {
       1.1.1.1
    }
}
检查配置文件没有问题。
然后怀疑问题出在/opt/check_nginx.sh 这个检测上面
/opt/check_nginx.sh 的内容是：
!/bin/bash
if [ $(ps -C nginx --no-header | wc -l) -eq 0 ]; then
     /etc/init.d/nginx  start
fi
sleep 2
if [ $(ps -C nginx --no-header | wc -l) -eq 0 ]; then
       /etc/init.d/keepalived stop
fi
仔细的读者会发现我这个脚本有问题，对!/bin/bash 前面少个#号。于是添加上，但是问题还是没有解决。 最后我再keepalive中取消了nginx的检测，keepalived正常了。
#!/bin/bash
if [ $(ps -C nginx --no-header | wc -l) -eq 0 ]; then
     /etc/init.d/nginx  start
fi
sleep 2
if [ $(ps -C nginx --no-header | wc -l) -eq 0 ]; then
       /etc/init.d/keepalived stop
fi
以上这个脚本是有问题的。下面是修改后正确的脚本：
#!/bin/bash
if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
then
 /usr/local/nginx/sbin/nginx
 sleep 5
 if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
 then
 killall keepalived
 fi
fi
后期补充：
日志中出现Process [6528] didn't respond to SIGTERM  这种问题check_nginx.sh也是这个脚本的问题，
大家注意一下。