高可用架构篇：【4】Keepalived + Nginx实现高可用Web负载均衡

一、场景需求

二、Keepalived简要介绍

• Keepalived 是一种高性能的服务器高可用或热备解决方案，Keepalived可以用来防止服务器单点故障的发生，通过配合Nginx可以实现web前端服务的高可用。
• Keepalived以VRRP协议为实现基础，用VRRP协议来实现高可用性(HA)。VRRP(Virtual Router Redundancy Protocol)协议是用于实现路由器冗余的协议，VRRP协议将两台或多台路由器设备虚拟成一个设备，对外提供虚拟路由器IP(一个或多个)，而在路由器组内部，如果实际拥有这个对外IP的路由器如果工作正常的话就是MASTER，或者是通过算法选举产生，MASTER实现针对虚拟路由器IP的各种网络功能，如ARP请求，ICMP，以及数据的转发等；其他设备不拥有该虚拟IP，状态是BACKUP，除了接收MASTER的VRRP状态通告信息外，不执行对外的网络功能。当主机失效时，BACKUP将接管原先MASTER的网络功能。
• VRRP协议使用多播数据来传输VRRP数据，VRRP数据使用特殊的虚拟源MAC地址发送数据而不是自身网卡的MAC地址，VRRP运行时只有MASTER路由器定时发送VRRP通告信息，表示MASTER工作正常以及虚拟路由器IP(组)，BACKUP只接收VRRP数据，不发送数据，如果一定时间内没有接收到MASTER的通告信息，各BACKUP将宣告自己成为MASTER，发送通告信息，重新进行MASTER选举状态。
• 

三、方案规划

• CentOS 6.6 x64
• keepalived-1.2.18.tar.gz
• nginx-1.6.2.tar.gz

VIP	IP	主机名	Nginx端口	默认主从
192.168.0.226	192.168.0.227	edu-proxy-01	88	MASTER
192.168.0.226	192.168.0.228	edu-proxy-02	88	MASTER

四、安装Nginx

1、安装编译Nginx所需的依赖包

• # yum install gcc gcc-c++ make automake autoconf libtool pcre pcre-devel zlib zlib-devel openssl openssl-devel

2、上传Nginx(nginx-1.6.2.tar.gz)到 /usr/local/src 目录

3、编译安装Nginx

# cd /usr/local/src/
# tar -zxvf nginx-1.6.2.tar.gz
# cd nginx-1.6.2
# ./configure --prefix=/usr/local/nginx
# make && make install

4、配置Nginx

• # vi /usr/local/nginx/conf/nginx.conf

user  root;
worker_processes  1;
#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;
#pid        logs/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';
    #access_log  logs/access.log  main;
    sendfile        on;
    #tcp_nopush     on;
    #keepalive_timeout  0;
    keepalive_timeout  65;
#gzip  on;
    server {
        listen       88;
        server_name  localhost;
        #charset koi8-r;
        #access_log  logs/host.access.log  main;
        location / {
            root   html;
            index  index.html index.htm;
        }
        #error_page  404              /404.html;
        # redirect server error pages to the static page /50x.html
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

• 修改Nginx欢迎首页内容（用于后面测试，用于区分两个节点的Nginx）：
  • # vi /usr/local/nginx/html/index.html
• 192.168.0.227中的标题加 1

  • Welcome to nginx! 1

• 192.168.0.228中的标题加 2

  • Welcome to nginx! 2

5、系统防火墙打开对应的端口88

• # vi /etc/sysconfig/iptables

## Nginx
-A INPUT -m state --state NEW -m tcp -p tcp --dport 88 -j ACCEPT

• # service iptables restart

6、测试Nginx是否安装成功

• # /usr/local/nginx/sbin/nginx -t

nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

7、Nginx命令

• 启动
  • # /usr/local/nginx/sbin/nginx
• 重启
  • # /usr/local/nginx/sbin/nginx -s reload

8、设置Nginx开机启动

• # vi /etc/rc.local
• 加入：
  • /usr/local/nginx/sbin/nginx

9、分别访问两个Nginx

五、安装Keepalived （ http://www.keepalived.org/download.html ）

1、上传或下载keepalived（keepalived-1.2.18.tar.gz）到 /usr/local/src 目录

2、解压安装

# cd /usr/local/src
# tar -zxvf keepalived-1.2.18.tar.gz
# cd keepalived-1.2.18
# ./configure --prefix=/usr/local/keepalived
# make && make install

3、将keepalived安装成Linux系统服务：

• 因为没有使用keepalived的默认路径安装（默认是/usr/local）,安装完成之后，需要做一些工作
  复制默认配置文件到默认路径

# mkdir /etc/keepalived
# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

• 复制keepalived服务脚本到默认的地址

# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
# ln -s /usr/local/sbin/keepalived /usr/sbin/
# ln -s /usr/local/keepalived/sbin/keepalived /sbin/

• 设置keepalived服务开机启动
  • # chkconfig keepalived on

4、修改Keepalived配置文件

(1) MASTER节点配置文件（192.168.0.227）

• # vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   ## keepalived自带的邮件提醒需要开启sendmail服务。建议用独立的监控或第三方SMTP
   router_id edu-proxy-01	## 标识本节点的字条串，通常为hostname
}
## keepalived会定时执行脚本并对脚本执行的结果进行分析，动态调整vrrp_instance的优先级。如果脚本执行结果为0，并且weight配置的值大于0，则优先级相应的增加。如果脚本执行结果非0，并且weight配置的值小于0，则优先级相应的减少。其他情况，维持原本配置的优先级，即配置文件中priority对应的值。
vrrp_script chk_nginx {
    script "/etc/keepalived/nginx_check.sh"	## 检测nginx状态的脚本路径
    interval 2	## 检测时间间隔
    weight -20	## 如果条件成立，权重-20 
}
## 定义虚拟路由，VI_1 为虚拟路由的标示符，自己定义名称
vrrp_instance VI_1 {
    state MASTER	## 主节点为MASTER，对应的备份节点为BACKUP
    interface eth1	## 绑定虚拟IP的网络接口，与本机IP地址所在的网络接口相同，我的是eth1
    virtual_router_id 51	## 虚拟路由的ID号，两个节点设置必须一样，可选IP最后一段使用, 相同的VRID为一个组，他将决定多播的MAC地址
    mcast_src_ip 192.168.0.227	## 本机IP地址
    priority 100	## 节点优先级，值范围0-254，MASTER要比BACKUP高
nopreempt	## 优先级高的设置nopreempt解决异常恢复后再次抢占的问题
advert_int 1	## 组播信息发送间隔，两个节点设置必须一样，默认1s
## 设置验证信息，两个节点必须一致
authentication {
        auth_type PASS
        auth_pass 1111  ## 真实生产，按需求对应该过来
}
## 将track_script块加入instance 配置块
    track_script {
        chk_nginx	## 执行Nginx监控的服务
}
## 虚拟IP池, 两个节点设置必须一样
    virtual_ipaddress {
        192.168.0.226	## 虚拟ip，可以定义多个
    }
}

(2)BACKUP节点配置文件（192.168.0.228）:

• # vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
   router_id edu-proxy-02
}

vrrp_script chk_nginx {
    script "/etc/keepalived/nginx_check.sh"
    interval 2
    weight -20
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth1
    virtual_router_id 51
    mcast_src_ip 192.168.0.228
    priority 70
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {
        chk_nginx
    }
    virtual_ipaddress {
        192.168.0.226
    }
}

5、编写Nginx状态检测脚本 /etc/keepalived/nginx_check.sh (已在keepalived.conf中配置)

• 脚本要求：如果nginx停止运行，尝试启动，如果无法启动则杀死本机的keepalived进程，keepalied将虚拟ip绑定到BACKUP机器上。内容如下：
• # vi /etc/keepalived/nginx_check.sh

#!/bin/bash
A=`ps -C nginx –no-header |wc -l`
if [ $A -eq 0 ];then
    /usr/local/nginx/sbin/nginx
    sleep 2
    if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then
        killall keepalived
    fi
fi

• 保存后，给脚本赋执行权限：
  • # chmod +x /etc/keepalived/nginx_check.sh

6、启动Keepalived

• # service keepalived start
  • Starting keepalived: [ OK ]

7、Keepalived+Nginx的高可用测试

(1)关闭192.168.0.227中的Nginx，Keepalived会将它重新启动

# /usr/local/nginx/sbin/nginx -s stop

(2)关闭192.168.0.227中的Keepalived，VIP会切换到192.168.0.228中

# service keepalived stop

• Keepalived停止后，该节点的网络接口中的VIP将消失
• 查看此时VIP对应的MAC，Windows下使用CMD命令查看：
• 
• 说明此时VIP已经漂移到物理主机192.168.0.228上了
• 再通过VIP来访问Nginx集群，访问到的也是192.168.0.228
• 

(3)重新启动192.168.0.227中的Keepalived，VIP又会切回到192.168.0.227中来

# service keepalived start

• 查看虚拟IP状态
  
• # ip add
  
• Keepalived启动后，网络接口上又会创建出VIP 192.168.0.226
• 
• Keepalived服务管理命令：
  • 停止：service keepalived stop
  • 启动：service keepalived start
  • 重启：service keepalived restart
  • 查看状态：service keepalived status

其他参考资料：

• keepalived之vrrp_script总结：http://my.oschina.net/hncscwc/blog/158746
• keepalived双机热备实现故障时发送邮件通知：http://www.2cto.com/os/201407/317795.html
• 基于keepalived 实现VIP转移，lvs，nginx的高可用：http://www.tuicool.com/articles/eu26Vz