Linux运维 第五阶段（十一）keepalived+{nginx,haproxy}
Linux运维 第五阶段（十一）keepalived+{nginx,haproxy}
环境：
[root@node1 ~]# uname -a
Linux node1.magedu.com2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013 x86_64 x86_64 x86_64GNU/Linux
准备：
VIP（192.168.41.222）
node1（nginx|haproxy’s master，192.168.41.133，安装nginx|haproxy和keepalived）
node2（nginx|haproxy’s backup，192.168.41.134，安装nginx|haproxy和keepalived）
node3（后端RS1，192.168.41.135，安装httpd）
node4（后端RS2，192.168.41.136，安装httpd）
注：node{1,2}要高可用，配置双机互信，时间同步
1、keepalived+nginx（nginx的安装参考《第三阶段（十五）理解LNMP》）：
node{1,2}-side：
[root@node1 ~]# yum -y groupinstall “Desktop Platform” “Desktop Platform Development” "Server Platform Development" “Development tools” “Compatibility libraries”（将这几个开发平台和兼容库安装上，防止编译时依赖某个库文件还要单独安装）
[root@node1 ~]# tar xf keepalived-1.2.19.tar.gz
[root@node1 ~]# cd keepalived-1.2.19
[root@node1 keepalived-1.2.19]# ./configure --help
[root@node1 keepalived-1.2.19]# ./configure --prefix=/usr/local/keepalived
[root@node1 keepalived-1.2.19]# make && make install
[root@node1 keepalived-1.2.19]# cd /usr/local/keepalived/
[root@node1 keepalived]# ls
bin etc sbin share
[root@node1 keepalived]# cp bin/genhash /bin
[root@node1 keepalived]# cp etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
[root@node1 keepalived]# cp etc/sysconfig/keepalived /etc/sysconfig/
[root@node1 keepalived]# mkdir /etc/keepalived
[root@node1 keepalived]# cp -r etc/keepalived/* /etc/keepalived/
[root@node1 keepalived]# cp sbin/keepalived /sbin/
[root@node1 keepalived]# cd
[root@node1 ~]# vim /etc/man.config（添加一行，这样可以直接使用#man keepalived.conf，否则要指定路径#man -M /usr/local/keepalived/share/man keepalived.conf）
MANPATH /usr/local/keepalived/share/man
[root@node1 ~]# . !$
注：编译安装时可使用#./configure --sysconfidr=/etc --bindir=/bin --sbindir=/sbin--mandir=/usr --prefix=/usr/local/keepalived这样在安装完成后省去以上复制的几步
[root@node1 ~]# chkconfig --add keepalived
[root@node1 ~]# chkconfig keepalived on
[root@node1 ~]# chkconfig --list keepalived
keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off
node1-side：
[root@node1 ~]# cd /etc/keepalived/
[root@node1 keepalived]# cp keepalived.conf keepalived.conf.bak
[root@node1 keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
mcast_src_ip 192.168.41.133
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.41.222/32 dev eth0 label eth0:0
}
}
[root@node1 keepalived]# scp keepalived.conf node2:/etc/keepalived/
keepalived.conf 100% 485 0.5KB/s 00:00
node2-side：
[root@node2 keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id51
mcast_src_ip 192.168.41.134
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.41.222/32 dev eth0 label eth0:0
}
}
注：mcast_src_ip <IPADDR>（绑定当前node地址，多播方式。default IP for binding vrrpd is the primary IP on interface. If you want to hide location of vrrpd, use this IP as src_addr for multicast or unicast vrrp packets. (since it’s multicast, vrrpd will get the reply packet no matter what src_addr is used).）
node{1,2}-side：
[root@node1 ~]# vim /etc/nginx/nginx.conf
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream websrvs {
server 192.168.41.135 weight=1 max_fails=2 fail_timeout=2;
server 192.168.41.136 weight=1 max_fails=2 fail_timeout=2;
server 127.0.0.1:8080 backup;
}
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://websrvs;
proxy_set_header X-Real-IP $remote_addr;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
server {
listen 8080;
server_name localhost;
location / {
root /web/errorpages;
index index.html;
}
}
}
[root@node1 ~]# cat /web/errorpages/index.html
Sorry,the server is maintaining
node{3,4}-side：
[root@node3 ~]# service httpd status
httpd (pid 2332) 正在运行...
[root@node4 ~]# service httpd status
httpd (pid 12734) 正在运行...
node-side：
[root@node1 ~]# elinks -dump http://192.168.41.135
RS1.magedu.com
[root@node1 ~]# elinks -dump http://192.168.41.136
RS2.magedu.com
[root@node1 ~]# service nginx start
正在启动 nginx： [确定]
[root@node1 ~]# service keepalived start
正在启动 keepalived： [确定]
node2-side：
[root@node2 ~]# elinks -dump http://192.168.41.136
RS2.magedu.com
[root@node2 ~]# elinks -dump http://192.168.41.135
RS1.magedu.com
[root@node2 ~]# service nginx start
正在启动 nginx： [确定]
[root@node2 ~]# service keepalived start
正在启动 keepalived： [确定]
测试：
再次刷新后
[root@node1 ~]# ifconfig eth0:0（由此可知当前是node1是活动状态）
eth0:0 Link encap:Ethernet HWaddr00:0C:29:E2:18:0E
inet addr:192.168.41.222 Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[root@node2 ~]# ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr00:0C:29:CC:D9:CD
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[root@node3 ~]# service httpd stop
停止httpd： [确定]
[root@node4 ~]# service httpd stop
停止 httpd： [确定]
再次刷新页面
[root@node3 ~]# service httpd start
正在启动 httpd： [确定]
[root@node4 ~]# service httpd start
正在启动 httpd： [确定]
[root@node1 ~]# service keepalived stop
停止 keepalived： [确定]
再次访问网页正常
[root@node1 ~]# ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr00:0C:29:E2:18:0E
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[root@node2 ~]# ifconfig eth0:0（将node1上的keepalived停掉，切换至node2，可正常提供服务）
eth0:0 Link encap:Ethernet HWaddr00:0C:29:CC:D9:CD
inet addr:192.168.41.222 Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
以上仅实现当node1或node2主机故障（或网络故障，或keepalived服务故障）时，可自动切换，但在nginx服务故障时并不能检测到，由此进一步配置
[root@node1 ~]# service keepalived start
正在启动 keepalived： [确定]
node{1,2}-side：
[root@node1 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script nginx_check {
script "[[ `ps -C nginx --no-header` -eq 0 ]] && exit 1 || exit 0"
interval 1
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
mcast_src_ip 192.168.41.133
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.41.222/32 dev eth0 label eth0:0
}
track_script {
nginx_check
}
}
这样能实现在node1（master）上的nginx服务down掉或者故障后会自动转移到node2（backup）
2、keepalived+haproxy：
node{1,2}-side：
[root@node1 ~]# vim /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen stats
mode http
bind 0.0.0.0:1080
stats enable
stats hide-version
stats uri /haproxyadmin?stats
stats realm Haproxy\ Statistics
stats auth admin:admin
stats admin if TRUE
frontend http-in
bind *:80
mode http
log global
option httpclose
option logasap
option dontlognull
capture request header Host len20
capture request header Refererlen 60
default_backend servers
backend servers
balance roundrobin
server websrv1 192.168.41.135:80 check maxconn 2000
server websrv2 192.168.41.136:80 check maxconn 2000
node1-side：
[root@node1 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script haproxy_check {
script "/etc/keepalived/haproxy_check.sh"
interval 1
weight 5
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
mcast_src_ip 192.168.41.133
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.41.222/32 dev eth0 label eth0:0
}
track_script{
haproxy_check
}
}
node2-side：
[root@node2 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script haproxy_check {
script "/etc/keepalived/haproxy_check.sh"
interval 1
weight 5
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
mcast_src_ip 192.168.41.134
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.41.222/32 dev eth0 label eth0:0
}
track_script{
haproxy_check
}
}
node{1,2}-side：
[root@node1 ~]# vim /etc/keepalived/haproxy_check.sh
#!/bin/bash
#
if [ `ps -C haproxy --no-header | wc -l` -eq 0 ] ; then
service haproxy start
fi
if [ `ps -C haproxy --no-header | wc -l` -eq 0 ] ; then
service keepalived stop
fi
[root@node1 ~]# chmod +x !$
[root@node1 ~]# scp /etc/keepalived/haproxy_check.sh node2:/etc/keepalived/
[root@node1 ~]# service haproxy start
正在启动 haproxy： [确定]
[root@node1 ~]# service keepalived start
正在启动 keepalived： [确定]
[root@node2 ~]# service haproxy start
正在启动 haproxy： [确定]
[root@node2 ~]# service keepalived start
正在启动 keepalived： [确定]
[root@node1 ~]# ifconfig eth0:0（当前活动节点在node1上）
eth0:0 Link encap:Ethernet HWaddr00:0C:29:E2:18:0E
inet addr:192.168.41.222 Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[root@node1 ~]# service haproxy stop（停止haproxy服务后，依照haproxy_check.sh脚本会再次启动它，如果仍无法启动时将会把当前活动node的keepalived服务停掉，停掉后会自动切换出去）
停止 haproxy： [确定]
[root@node1 ~]# service haproxy status
haproxy (pid 43028) 正在运行...
[root@node1 ~]# service keepalived stop（模拟keepalived服务停止后，切换至node2）
停止 keepalived： [确定]
[root@node1 ~]# ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr00:0C:29:E2:18:0E
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[root@node2 keepalived]# cd
[root@node2 ~]# ifconfig eth0:0
eth0:0 Link encap:Ethernet HWaddr00:0C:29:CC:D9:CD
inet addr:192.168.41.222 Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1