keepalived结合nginx主备高可用实现

说明：

    两台服务器ip:192.168.163.128(主)  192.168.163.129(备)

    两台服务器分别安装nginx和keepalived

安装配置：   

首先安装nginx（yum和编译安装都可以，128和129的安装方式一样，这里以128为例）：

如果没有服务器没有nginx的rpm包则需要先安装扩展源：

yum install -y epel-release

yum安装nginx ：

yum install -y nginx 

安装完成后查看nginx的rpm包：

[root@localhost ~]# rpm -qa nginx
nginx-1.10.2-1.el6.x86_64

查看nginx安装目录：

[root@localhost ~]# rpm -ql nginx
/etc/logrotate.d/nginx
/etc/nginx/conf.d/default.conf
/etc/nginx/conf.d/ssl.conf
/etc/nginx/conf.d/virtual.conf
/etc/nginx/fastcgi.conf
/etc/nginx/fastcgi.conf.default
/etc/nginx/fastcgi_params
/etc/nginx/fastcgi_params.default
/etc/nginx/koi-utf
/etc/nginx/koi-win
/etc/nginx/mime.types
/etc/nginx/mime.types.default
/etc/nginx/nginx.conf
/etc/nginx/nginx.conf.default
/etc/nginx/scgi_params
/etc/nginx/scgi_params.default
/etc/nginx/uwsgi_params
/etc/nginx/uwsgi_params.default
/etc/nginx/win-utf
/etc/rc.d/init.d/nginx
/etc/sysconfig/nginx
/usr/lib64/nginx/modules
/usr/sbin/nginx
/usr/share/doc/nginx-1.10.2
/usr/share/doc/nginx-1.10.2/CHANGES
/usr/share/doc/nginx-1.10.2/LICENSE
/usr/share/doc/nginx-1.10.2/README
/usr/share/doc/nginx-1.10.2/README.dynamic
/usr/share/doc/nginx-1.10.2/UPGRADE-NOTES-1.0-to-1.10
/usr/share/man/man3/nginx.3pm.gz
/usr/share/man/man8/nginx.8.gz
/usr/share/nginx/html/404.html
/usr/share/nginx/html/50x.html
/usr/share/nginx/html/index.html
/usr/share/nginx/html/nginx-logo.png
/usr/share/nginx/html/poweredby.png
/usr/share/vim/vimfiles/ftdetect/nginx.vim
/usr/share/vim/vimfiles/indent/nginx.vim
/usr/share/vim/vimfiles/syntax/nginx.vim
/var/lib/nginx
/var/lib/nginx/tmp
/var/log/nginx

主要关注nginx配置和html文件（标红部分）

nginx配置文件：

[root@localhost ~]# vim /etc/nginx/nginx.conf
user root;
worker_processes 1;
#error_log logs/error.log;  
#error_log logs/error.log notice;  
#error_log logs/error.log info;  
#pid logs/nginx.pid;  
events {
    worker_connections 1024;
}
http {
    include mime.types;
    default_type application/octet-stream;
    #log_format main '$remote_addr - $remote_user [$time_local] "$request" '  
    # '$status $body_bytes_sent "$http_referer" '  
    # '"$http_user_agent" "$http_x_forwarded_for"';  
    #access_log logs/access.log main;  
    sendfile on;
    #tcp_nopush on;  
    #keepalive_timeout 0;  
    keepalive_timeout 65;
    #gzip on;  
    server {
        listen 80;
        server_name localhost;
        #charset koi8-r;  
        #access_log logs/host.access.log main;  
        location / {
            root /usr/share/nginx/html;
            index index.html index.htm;
        }
        #error_page 404 /404.html;  
        # redirect server error pages to the static page /50x.html  
        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
            root html;
        }
    }
}

修改默认访问页内容：

192.168.163.128 的标题 
Welcome to nginx! 128
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.

Commercial support is available at
nginx.com.
Thank you for using nginx.

启动nginx：

service nginx start

查看nginx进程：

[root@localhost ~]# ps -ef|grep nginx
root      1811     1  0 19:41 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
root      1813  1811  0 19:41 ?        00:00:00 nginx: worker process                   
root     10026  1512  0 20:50 pts/0    00:00:00 vim /usr/share/nginx/html/index.html
root     10628  1529  0 20:55 pts/1    00:00:00 grep nginx

yum安装keepalived服务(主备安装方式相同)：

yum install -y keepalived 

查看安装路径：

[root@localhost ~]# rpm -ql keepalived
/etc/keepalived
/etc/keepalived/keepalived.conf
/etc/rc.d/init.d/keepalived
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/libexec/keepalived
/usr/sbin/keepalived
/usr/share/doc/keepalived-1.2.13
/usr/share/doc/keepalived-1.2.13/AUTHOR
/usr/share/doc/keepalived-1.2.13/CONTRIBUTORS
/usr/share/doc/keepalived-1.2.13/COPYING
/usr/share/doc/keepalived-1.2.13/ChangeLog
/usr/share/doc/keepalived-1.2.13/NOTE_vrrp_vmac.txt
/usr/share/doc/keepalived-1.2.13/README
/usr/share/doc/keepalived-1.2.13/TODO
/usr/share/doc/keepalived-1.2.13/VERSION
/usr/share/doc/keepalived-1.2.13/keepalived.conf.SYNOPSIS
/usr/share/doc/keepalived-1.2.13/samples
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.HTTP_GET.port
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.IPv6
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.SMTP_CHECK
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.SSL_GET
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.fwmark
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.inhibit
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.misc_check
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.misc_check_arg
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.quorum
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.sample
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.status_code
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.track_interface
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.virtual_server_group
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.virtualhost
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.vrrp
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.vrrp.localcheck
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.vrrp.lvs_syncd
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.vrrp.routes
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.vrrp.scripts
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.vrrp.static_ipaddress
/usr/share/doc/keepalived-1.2.13/samples/keepalived.conf.vrrp.sync
/usr/share/doc/keepalived-1.2.13/samples/sample.misccheck.smbcheck.sh
/usr/share/man/man1/genhash.1.gz
/usr/share/man/man5/keepalived.conf.5.gz
/usr/share/man/man8/keepalived.8.gz
/usr/share/snmp/mibs/KEEPALIVED-MIB.txt

配置文件路径：/etc/keepalived/keepalived.conf

清空配置文件：

> /etc/keepalived/keepalived.conf

主服务器的配置文件：

global_defs { ###全局定义参数
   notification_email { ###出现问题时给邮箱发邮件
     [email protected] ###由于本实验用不到发邮件的功能 所以可以不用定义 或者随便写一下
   }
   notification_email_from [email protected] ###由哪一个邮箱发出去  可以引用第三方
   smtp_server 127.0.0.1  ###sendmail 服务
   smtp_connect_timeout 30  ###超时时间 s
   router_id zhang129 ##### 标识本节点的字条串，通常为 hostname
}
vrrp_script chk_nginx {  ### 执行脚本 检测nginx服务是否正常
    script "/usr/local/check_ng.sh" ###定义检查脚本路径并编写检查脚本来检查这个服务是否正常 如果不正常  要把它启动起来   名字可以自己定义  脚本名字和这里定义的保持一直就行
    interval 3 ###检测的间断是3秒钟
    weight -20 ###权重 
}
vrrp_instance VI_1 {
    state MASTER ## 主节点为 MASTER， 对应的备份节点为 BACKUP
    interface eth0 ## 绑定虚拟 IP 的网络接口，与本机 IP 地址所在的网络接口相同， eth0 
    virtual_router_id 51 ## 虚拟路由的 ID 号， 两个节点设置必须一样， 可选 IP 最后一段使用, 相同的 VRID 为一个组，这将决定多播的 MAC 地址  
    mcast_src_ip 192.168.163.129  ## 本机 IP 地址
    nopreempt ## 优先级高的设置 nopreempt 解决异常恢复后再次抢占的问题
    priority 100 ## 节点优先级， 值范围 0-254， MASTER 要比 BACKUP 高
    advert_int 1 ## 组播信息发送间隔，两个节点设置必须一样， 默认 1s
    authentication { ##设置验证信息 两个节点必须设置一致
        auth_type PASS ##密码类型
        auth_pass zhang ## 密码 ，真实生产，按需求对应该过来 
    } 
    virtual_ipaddress {
        192.168.163.130 # 虚拟 IP 池, 两个节点设置必须一样 即vip 可以定义多个 
    } 
    track_script { ## 将     track_script 块加入 instance 配置块 
        chk_nginx ## 执行 Nginx 监控的服务 
    }
}

check_ng.sh脚本：

[root@localhost ~]# vim /usr/local/check_ng.sh 
#!/bin/bash
d=`date --date today +%Y%m%d_%H:%M:%S`
#计算nginx进程数量
n=`ps -C nginx --no-heading|wc -l`
#如果进程为0，则启动nginx，并且再次检测nginx进程数量，
#如果还为0，说明nginx无法启动，此时需要关闭keepalived
if [ $n -eq 0 ]; then
        service nginx start
        n2=`ps -C nginx --no-heading|wc -l`
        if [ $n2 -eq 0  ]; then
                echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
                service keepalived stop
        fi
fi
   

此处注意脚本中的`  ,不是英文'

命令替换.`command` 结构使字符(`) [译者注：这个字符不是单引号，而是在标准美国键盘上的ESC键下面，在字符1左边，在TAB键上面的那个键，要特别留心] 引住的命令（command）执行结果能赋值给一个变量。它也被称为后引号（backquotes）或是斜引号（backticks）

添加脚本执行权限：chmod 755 /usr/local/check_ng.sh

到此 ，主服务器的nginx和keepalived已经安装完毕；

备服务器的nginx和keepalived安装同主服务器安装方法

注意修改nginx的默认访问页面/usr/share/nginx/html/index.html 内容： 添加 备用服务器的ip，以便查看测试结果

备用keepalived的配置文件/etc/keepalived/keepalived.conf：

[root@localhost ~]# vim /etc/keepalived/keepalived.conf 
global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id zhang129
}
vrrp_script chk_nginx {
    script "/usr/local/check_ng.sh"
    interval 3
    weight -20
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    mcast_src_ip 192.168.163.129
    nopreempt
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass zhang
    }
    virtual_ipaddress {
        192.168.163.130
    }
    track_script {
        chk_nginx
    }
}

监控脚本/usr/local/check_ng.sh同主服务器一致；

此时，主备服务器的nginx和keepalived都已安装完成

------------------------------------------------------------------------------

测试：

首先开启主服务器的nginx：

service nginx start

访问主服务器nginx：

开启主服务器的keepalived：

service keepalived start

查看keepalived日志：

[root@localhost local]# tail -f /var/log/messages
Apr 28 19:43:36 localhost Keepalived_vrrp[7084]: Using LinkWatch kernel netlink reflector...
Apr 28 19:43:36 localhost Keepalived_vrrp[7084]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Apr 28 19:43:36 localhost Keepalived_vrrp[7084]: VRRP_Script(chk_nginx) succeeded  ####脚本运行成功
Apr 28 19:43:36 localhost Keepalived_vrrp[7084]: VRRP_Instance(VI_1) Transition to MASTER STATE
Apr 28 19:43:36 localhost Keepalived_vrrp[7084]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
Apr 28 19:43:37 localhost Keepalived_vrrp[7084]: VRRP_Instance(VI_1) Entering MASTER STATE
Apr 28 19:43:37 localhost Keepalived_vrrp[7084]: VRRP_Instance(VI_1) setting protocol VIPs.
Apr 28 19:43:37 localhost Keepalived_vrrp[7084]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.163.130
Apr 28 19:43:37 localhost Keepalived_healthcheckers[7083]: Netlink reflector reports IP 192.168.163.130 added
Apr 28 19:43:42 localhost Keepalived_vrrp[7084]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.163.130

关闭主服务器的nginx，查看nginx进程发现依然有nginx进程，因为keepalived监控脚本执行重启了nginx

[root@localhost ~]# !ps
ps -ef|grep nginx
root      1811     1  0 19:41 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
root      1813  1811  0 19:41 ?        00:00:00 nginx: worker process                   
root     10026  1512  0 20:50 pts/0    00:00:00 vim /usr/share/nginx/html/index.html
root     19095  1529  0 22:06 pts/1    00:00:00 grep nginx
[root@localhost ~]# ps -ef|grep keepalived
root      1834     1  0 19:42 ?        00:00:00 /usr/sbin/keepalived -D
root      1835  1834  0 19:42 ?        00:00:00 /usr/sbin/keepalived -D
root      1837  1834  0 19:42 ?        00:00:04 /usr/sbin/keepalived -D
root     19115  1529  0 22:06 pts/1    00:00:00 grep keepalived
[root@localhost ~]# service nginx stop
停止 nginx：                                               [确定]
[root@localhost ~]# ps -ef|grep keepalived
root      1834     1  0 19:42 ?        00:00:00 /usr/sbin/keepalived -D
root      1835  1834  0 19:42 ?        00:00:00 /usr/sbin/keepalived -D
root      1837  1834  0 19:42 ?        00:00:04 /usr/sbin/keepalived -D
root     19194  1529  0 22:06 pts/1    00:00:00 grep keepalived
[root@localhost ~]# ps -ef|grep nginx
root     10026  1512  0 20:50 pts/0    00:00:00 vim /usr/share/nginx/html/index.html
root     19187     1  0 22:06 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
root     19189 19187  0 22:06 ?        00:00:00 nginx: worker process                   
root     19202  1529  0 22:06 pts/1    00:00:00 grep nginx

此时访问vip地址，发现访问到128(主服务器)的nginx，说明keepalived配置成功；

开启备用服务器nginx：

service nginx start

访问备用服务器下的nginx的ip：

开启备用服务器的keepalived：

service keepalived start

访问vip：

关闭主服务器的keepalived服务，查看主服务器的keepalived日志：

[root@localhost ~]# service keepalived stop
停止 keepalived：                                          [确定]
[root@localhost ~]# 
[root@localhost local]# tail -f /var/log/messages
Apr 28 22:22:25 localhost Keepalived[7081]: Stopping Keepalived v1.2.13 (03/19,2015)
Apr 28 22:22:25 localhost Keepalived_vrrp[7084]: VRRP_Instance(VI_1) sending 0 priority
Apr 28 22:22:25 localhost Keepalived_vrrp[7084]: VRRP_Instance(VI_1) removing protocol VIPs.
Apr 28 22:22:25 localhost Keepalived_healthcheckers[7083]: Netlink reflector reports IP 192.168.163.130 removed

可以看出 vip在主服务器的keepalived绑定移除了；

此时查看备用服务器的keepalived日志:

[root@localhost ~]# !tail
tail -f /var/log/messages 
Apr 28 22:22:21 localhost Keepalived_vrrp[1837]: VRRP_Instance(VI_1) Transition to MASTER STATE
Apr 28 22:22:22 localhost Keepalived_vrrp[1837]: VRRP_Instance(VI_1) Entering MASTER STATE
Apr 28 22:22:22 localhost Keepalived_vrrp[1837]: VRRP_Instance(VI_1) setting protocol VIPs.
Apr 28 22:22:22 localhost Keepalived_healthcheckers[1835]: Netlink reflector reports IP 192.168.163.130 added
Apr 28 22:22:22 localhost Keepalived_vrrp[1837]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.163.130
Apr 28 22:22:27 localhost Keepalived_vrrp[1837]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.163.130

发现备用服务器的keepalived绑定了vip

访问vip：

访问到了129的nginx，说明主备切换成功；

此时关闭主服务器的nginx，开启主服务器的keepalived：

service keepalived start

查看nginx进程，发现nginx有启动了；

查看主服务器keepalived日志，发现主服务器keepalived又绑定了vip：

[root@localhost local]# tail -f /var/log/messages
Apr 28 22:22:25 localhost Keepalived_healthcheckers[7083]: Netlink reflector reports IP 192.168.163.130 removed
Apr 28 22:29:40 localhost Keepalived[26222]: Starting Keepalived v1.2.13 (03/19,2015)
Apr 28 22:29:40 localhost Keepalived[26223]: Starting Healthcheck child process, pid=26225
Apr 28 22:29:40 localhost Keepalived[26223]: Starting VRRP child process, pid=26226
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Netlink reflector reports IP 192.168.163.128 added
Apr 28 22:29:40 localhost Keepalived_healthcheckers[26225]: Netlink reflector reports IP 192.168.163.128 added
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Netlink reflector reports IP 172.17.42.1 added
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Netlink reflector reports IP fe80::20c:29ff:fef6:e5f0 added
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Netlink reflector reports IP fe80::2ccf:b6ff:fe1c:2abd added
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Registering Kernel netlink reflector
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Registering Kernel netlink command channel
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Registering gratuitous ARP shared channel
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Opening file '/etc/keepalived/keepalived.conf'.
Apr 28 22:29:40 localhost Keepalived_healthcheckers[26225]: Netlink reflector reports IP 172.17.42.1 added
Apr 28 22:29:40 localhost Keepalived_healthcheckers[26225]: Netlink reflector reports IP fe80::20c:29ff:fef6:e5f0 added
Apr 28 22:29:40 localhost Keepalived_healthcheckers[26225]: Netlink reflector reports IP fe80::2ccf:b6ff:fe1c:2abd added
Apr 28 22:29:40 localhost Keepalived_healthcheckers[26225]: Registering Kernel netlink reflector
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Truncating auth_pass to 8 characters
Apr 28 22:29:40 localhost Keepalived_healthcheckers[26225]: Registering Kernel netlink command channel
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Configuration is using : 64123 Bytes
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: Using LinkWatch kernel netlink reflector...
Apr 28 22:29:40 localhost Keepalived_healthcheckers[26225]: Opening file '/etc/keepalived/keepalived.conf'.
Apr 28 22:29:40 localhost Keepalived_healthcheckers[26225]: Configuration is using : 6433 Bytes
Apr 28 22:29:40 localhost Keepalived_healthcheckers[26225]: Using LinkWatch kernel netlink reflector...
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: VRRP_Script(chk_nginx) succeeded
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: VRRP_Instance(VI_1) Transition to MASTER STATE
Apr 28 22:29:40 localhost Keepalived_vrrp[26226]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
Apr 28 22:29:41 localhost Keepalived_vrrp[26226]: VRRP_Instance(VI_1) Entering MASTER STATE
Apr 28 22:29:41 localhost Keepalived_vrrp[26226]: VRRP_Instance(VI_1) setting protocol VIPs.
Apr 28 22:29:41 localhost Keepalived_vrrp[26226]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.163.130
Apr 28 22:29:41 localhost Keepalived_healthcheckers[26225]: Netlink reflector reports IP 192.168.163.130 added
Apr 28 22:29:46 localhost Keepalived_vrrp[26226]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.163.130

此时访问vip：

切换成功！！！！！

到此keepalived高可用搭建测试完成！！！

keepalived启动后查看vip绑定：

[root@localhost local]# ip add
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:f6:e5:f0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.163.128/24 brd 192.168.163.255 scope global eth0
    inet 
    inet 192.168.163.130/32 scope global eth0
    inet6 fe80::20c:29ff:fef6:e5f0/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0:  mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 2e:cf:b6:1c:2a:bd brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
    inet6 fe80::2ccf:b6ff:fe1c:2abd/64 scope link 
       valid_lft forever preferred_lft forever