方案背景介绍
根据公司业务需求，需要保证对外提供服务的web服务具有高可用及负载均衡的特性。
此方案省略部署web服务相关步骤。
环境介绍
| 名称 | IP地址 | 操作系统 |
|---|---|---|
| web1 | 192.168.1.231 | CentOS 6.5 |
| web2 | 192.168.1.239 | CentOS 6.5 |
| nginx1（主） | 192.168.1.233 | CentOS 6.5 |
| nginx2（备） | 192.168.1.241 | CentOS 6.5 |
nginx版本：1.10.2
keepalived版本：1.2.20
VIP（虚拟IP）：192.168.1.230
实施步骤
前期准备
每台Nginx服务器安装相关服务
yum install gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel
安装并配置Nginx
安装Nginx
两台Nginx服务器均安装Nginx 1.10.2
[root@nginx1 ~]# mkdir /usr/local/nginx
[root@nginx1 ~]# cd /usr/local/src/
[root@nginx1 src]# ls
nginx-1.10.2.tar.gz
[root@nginx1 src]# tar zxvf nginx-1.10.2.tar.gz
[root@nginx1 src]# cd nginx-1.10.2
[root@nginx1 nginx-1.10.2]# ./configure --prefix=/usr/local/nginx --with-http_ssl_module
[root@nginx1 nginx-1.10.2]# make
[root@nginx1 nginx-1.10.2]# make install
配置Nginx负载均衡
两台Nginx均如下配置，进行负载均衡。
注：根据实际情况，可以不进行负载均衡，而仅进行反向代理。
[root@nginx1 ~]# cd /usr/local/nginx/conf/
[root@nginx1 conf]# vi nginx.conf
http {
... ...
# 将需要负载均衡服务放入域中
upstream zone_test {
ip_hash;
server 192.168.1.231:8888;
server 192.168.1.239:7777;
}
server {
listen 80;
server_name localhost;
... ...
# 此处'http://'后需跟之前创建的域
location / {
proxy_pass http://zone_test;
}
... ...
}
安装并配置keepalived
安装keepalived
两台Nginx服务器均安装keepalived 1.2.20
[root@nginx1 ~]# mkdir /usr/local/keepalived
[root@nginx1 ~]# cd /usr/local/src/
[root@nginx1 src]# tar zxvf keepalived-1.2.20.tar.gz
[root@nginx1 src]# cd keepalived-1.2.20
[root@nginx1 keepalived-1.2.20]# ./configure --prefix=/usr/local/keepalived/
[root@nginx1 keepalived-1.2.20]# make && make install
做成服务模式，方便启动和关闭
将相应文件拷贝至系统目录，做成服务模式；并将keepalived设置成开机自启
# 源码安装
[root@nginx1 ~]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
[root@nginx1 ~]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@nginx1 ~]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@nginx1 ~]# cp -r /usr/local/keepalived/etc/keepalived/ /etc/
# 设置开机自启
[root@nginx1 ~]# chkconfig --add keepalived
[root@nginx1 ~]# chkconfig keepalived on
[root@nginx1 ~]# chkconfig --list keepalived
keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off
编写检查Nginx是否存活脚本
两台Nginx服务器上均需编写
[root@nginx1 ~]# mkdir /root/bin
[root@nginx1 ~]# cd /root/bin/
[root@nginx1 bin]# vi check_nginx.sh
以下是脚本内容
#!/bin/sh
# Filename: check_nginx.sh
# Edited By Yiji Kong
# This script checks nginx process which is alived or not
# 定义别名
alias PS='ps aux | grep nginx | grep -v grep | grep -v check_nginx | wc -l'
# 将nginx的绝对路径定义为NGINX
NGINX=/usr/local/nginx/sbin/nginx
if [ `PS` -gt 0 ]; then # 如果命令执行结果大于0，说明Nginx正在运行，则返回正常(0)
exit 0
# 否则重启一次Nginx后再次检测，还启动不了则关闭keepalived
else
${NGINX}
sleep 3
if [ `PS` -eq 0 ]; then
killall -9 keepalived
fi
fi
为脚本添加执行权限
[root@nginx1 bin]# chmod +x check_nginx.sh
修改两台Nginx服务器上keepalived的配置文件
配置文件位置：
/etc/keepalived/keepalived.conf
- 主Nginx服务器的keepalived配置文件如下：
! Configuration File for keepalived
global_defs {
notification_email {
#[email protected]
#[email protected]
#[email protected]
}
#notification_email_from [email protected]
#smtp_server 192.168.200.1
#smtp_connect_timeout 30
router_id LVS_DEVEL # router_id可以不同
#vrrp_skip_check_adv_addr
#vrrp_strict
}
# 定义检测脚本
vrrp_script check_nginx {
script "/root/bin/check_nginx.sh" # 脚本位置
interval 2 # 间隔，单位为秒
weight -2 # 权重-2
}
vrrp_instance VI_1 {
state MASTER # 实例的初始状态
interface eth0 # 监控VIP绑定的实际网卡
virtual_router_id 51 # 相同组的VRID需相同
priority 100 # 初始权重值
advert_int 1 # 组播信息发送间隔，单位为秒
authentication {
auth_type PASS # 启用密码验证
auth_pass 1111 # 密码，相同组的需要一致
}
# 执行检测脚本
track_script {
check_nginx
}
# 设置VIP地址，可以多个
virtual_ipaddress {
192.168.1.230
}
}
- 备Nginx服务器的keepalived配置文件如下：
! Configuration File for keepalived
global_defs {
notification_email {
#[email protected]
#[email protected]
#[email protected]
}
#notification_email_from [email protected]
#smtp_server 192.168.200.1
#smtp_connect_timeout 30
router_id LVS_DEVEL # router_id可以不同
#vrrp_skip_check_adv_addr
#vrrp_strict
}
# 定义检测脚本
vrrp_script check_nginx {
script "/root/bin/check_nginx.sh" # 脚本位置
interval 2 # 间隔，单位为秒
weight -2 # 权重-2
}
vrrp_instance VI_1 {
state BACKUP # 实例的初始状态
interface eth0 # 监控VIP绑定的实际网卡
virtual_router_id 51 # 相同组的VRID需相同
priority 99 # 初始权重值
advert_int 1 # 组播信息发送间隔，单位为秒
authentication {
auth_type PASS # 启用密码验证
auth_pass 1111 # 密码，相同组的需要一致
}
# 执行检测脚本
track_script {
check_nginx
}
# 设置VIP地址，可以多个
virtual_ipaddress {
192.168.1.230
}
}
至此，Nginx和keepalived的部署已经完成
绑定虚拟IP（VIP）
在启动keepalived前，需先在两台Nginx服务器上绑定VIP，否则无法对外提供服务。
有两种方法可以绑定，分别为临时和永久
- 第一种方法（临时）：
[root@nginx1 ~]# ifconfig eth0:1 192.168.1.230 broadcast 192.168.1.255 netmask 255.255.255.0 up
[root@nginx1 ~]# route add -host 192.168.1.230 dev eth0:1
- 第二种方法（永久）：
[root@nginx1 ~]# cd /etc/sysconfig/network-scripts/
[root@nginx1 network-scripts]# vi ifcfg-eth0:0
# 以下是"ifcfg-eth0:0"的详细配置
DEVICE=eth0:0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.1.230
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
# 重启网络服务
[root@nginx1 network-scripts]# service network restart
Shutting down interface eth0: Device state: 3 (disconnected)
[ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth0: Active connection state: activated
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/2
[ OK ]
# 查看VIP是否生效
[root@nginx1 network-scripts]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:15:5D:01:CA:11
inet addr:192.168.1.233 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::215:5dff:fe01:ca11/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4025628 errors:0 dropped:0 overruns:0 frame:0
TX packets:4960923 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4762501119 (4.4 GiB) TX bytes:4820676229 (4.4 GiB)
eth0:0 Link encap:Ethernet HWaddr 00:15:5D:01:CA:11
inet addr:192.168.1.230 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:3554 errors:0 dropped:0 overruns:0 frame:0
TX packets:3554 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:285168 (278.4 KiB) TX bytes:285168 (278.4 KiB)
启动keepalived
至此，可以启动keepalived了。
另外，无需事先启动Nginx，因为检查Nginx的脚本已经自带拉起的功能
[root@nginx2 sbin]# service keepalived start
Starting keepalived: [ OK ]
验证
- 模拟其中一台的Nginx突然关闭，看是否能自动拉起
- 模拟其中一台的keepalived突然关闭，看是否能自动切换
- 模拟其中一台网卡突然失效，看是否能够自动切换
- 模拟其中一台突然宕机，看是否能自动切换
具体步骤略
参考资料
- 使用Nginx+Keepalived组建高可用负载平衡Web server集群
- keepalived+nginx双机热备+负载均衡
- keepalived实践
- keepalived之vrrp_script总结