nginx+keepalived+tomcat配置高可用web集群



基本架构:


nginx+keepalived+tomcat配置高可用web集群_第1张图片


    

角色 ip 安装软件 作用 主机名
nginx主 192.168.247.129 nginx+keepalived 反向代理 nginxmaster.com
nginx备 192.168.247.130 nginx+keepalived 反向代理 nginxsalve.com
tomcat1 192.168.247.128 tomcat web服务器 tomcat1.com
tomcat2 192.168.247.131 tomcat web服务器 tomcat2.com
nfs主 192.168.247.132 nfs 低端共享存储 nfsmaster.com
nfs备 192.168.247.133 nfs 低端共享存储 nfsslave.com
虚拟ip 192.168.247.150






集群部署:

    tomcat安装部署:

        编译安装tomcat:(tomcat1,tomcat2)

            安装JDK,运行java环境:(tomcat1,tomcat2)

                卸载centos自带的JDK:

rpm -qa | grep java
yum -y remove java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5

                下载安装JDK:

##下载地址:http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html 
##下载最新的(64位):jdk-7u79-linux-x64.rpm
##在/usr下建立一个java目录,以备将java程序安装在此目录下
mkdir /usr/java
cd /usr/local/src
rpm -ivh jdk-7u79-linux-x64.rpm        ##安装后,在/usr/java目录下就会生成一个新的目录jdk1.7.0_79,该目录下存放的是安装后的文件
##设置环境变量:
vi /etc/profile  ##在最后添加:
export JAVA_HOME=/usr/java/jdk1.7.0_79
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

source /etc/profile
##验证是否安装成功:
java -version         ##出现如下则正常
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)


            安装tomcat7:(tomcat1,tomcat2)

软件:apache-tomcat-7.0.29.tar.gz
cd /usr/local/src/
tar -zxf apache-tomcat-7.0.42.tar.gz
cp -a apache-tomcat-7.0.42 /usr/local/tomcat7/
/usr/local/tomcat7/bin/startup.sh       ##启动tomcat,出现如下则正常
Using CATALINA_BASE:   /usr/local/tomcat7
Using CATALINA_HOME:   /usr/local/tomcat7
Using CATALINA_TMPDIR: /usr/local/tomcat7/temp
Using JRE_HOME:        /usr/java/jdk1.7.0_79
Using CLASSPATH:       /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar

##设置开机自动启动
##访问下:    ip:8080   看是否正常

        配置tomcat:(tomcat1,tomcat2)

            配置server.xml文件:

cd /usr/local/tomcat7/conf/
vi server.xml
##找到:
    
##将其注释掉

##添加如下(已优化):
    
##配置优化: 
            优化JVM堆内存:
                vi /usr/local/tomcat7/bin/catalina.sh
                添加:JAVA_OPTS='-server -Xms1024m -Xmx2048m -XX:PermSize=256M -XX:MaxNewSize=256m -XX:MaxPermSize=256m'
		
	    修改日志目录:/data/tomcat7/logs
	        vi /usr/local/tomcat7/conf/server.xml
			            #nginx做反向代理,%{X-Real-IP}i 获取用户的真实ip              

                
                    ##找到:Valve标签
                    ##将pattern参数的值修改为:"%{X-Real-IP}i %l %u %t "%r" %s %b"        ##{X-Real-IP}i,获取的是真实客户端ip,不是代理机ip
                    ##每个参数的含义:http://twb.iteye.com/blog/182100
                    ##nginx前端代理tomcat取真实客户端IP:http://www.cnphp6.com/archives/60908
##重新启动tomcat:
[root@tomcat1 ~]# /usr/local/tomcat7/bin/shutdown.sh 
Using CATALINA_BASE:   /usr/local/tomcat7
Using CATALINA_HOME:   /usr/local/tomcat7
Using CATALINA_TMPDIR: /usr/local/tomcat7/temp
Using JRE_HOME:        /usr/java/jdk1.7.0_79
Using CLASSPATH:       /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar
[root@tomcat1 ~]# /usr/local/tomcat7/bin/startup.sh 
Using CATALINA_BASE:   /usr/local/tomcat7
Using CATALINA_HOME:   /usr/local/tomcat7
Using CATALINA_TMPDIR: /usr/local/tomcat7/temp
Using JRE_HOME:        /usr/java/jdk1.7.0_79
Using CLASSPATH:       /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar
##tomcat性能优化:
参考:http://www.cnblogs.com/ggjucheng/archive/2013/04/16/3024731.html




    反向代理的部署:

        编译安装nginx:(主,备)

            安装nginx的依赖包:

yum -y install gcc gcc-c++ autoconf automake
yum -y install zlib zlib-devel openssl openssl-devel pcre-devel

            编译安装pcre,使nginx支持rewrite重写:

cd /usr/local/src/
wget http://sourceforge.net/projects/pcre/files/pcre/8.35/pcre-8.35.tar.gz
tar -zxf pcre-8.35.tar.gz
cd pcre-8.35
./configure
make
make install

            安装nginx:

cd /usr/local/src/
wget http://nginx.org/download/nginx-1.6.2.tar.gz
tar -zxf nginx-1.6.2.tar.gz 
cd nginx-1.6.2
./configure  --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_gzip_static_module --with-http_flv_module --with-http_gunzip_module --with-http_realip_module
make
make install
###注意--with-http_ssl_module,后面要用ssl

        配置nginx:(主,备)

            创建nginx用户:

useradd -s /sbin/nologin nginx

            修改文件打开数:

ulimit -n 65535
##设置开机自动启动

            生成ssl证书(模拟)(忽略)

##创建证书存放目录:
cd /usr/local/nginx/
mkdir ca
cd ca
##创建服务器私钥:
openssl genrsa -des3 -out server.key 1024
##创建签名请求的证书(CSR):
openssl req -new -key server.key -out server.csr
##在加载SSL支持的Nginx并使用上述私钥时除去必须的口令
cp server.key server.key.org
openssl rsa -in server.key.org -out server.key
##最后标记证书使用上述私钥和CSR:
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

         生成ssl证书(模拟)

##创建证书存放目录:
mkdir /usr/local/nginx/ca
cd /usr/local/nginx/ca
##生成密钥文件key和csr文件:
openssl req -new -newkey rsa:2048 -nodes -keyout server.key -out server.csr
##生成服务器认证文件crt:
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

           修改配置文件:

# mv /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf.old
# vi /usr/local/nginx/conf/nginx.conf
user nginx nginx;
worker_processes  2;
error_log  logs/error.log;
pid        logs/nginx.pid;
worker_rlimit_nofile 65535;

events {
    use epoll;
    worker_connections  65535;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" $http_x_forwarded_for';
    
    server_names_hash_bucket_size 128;
    client_header_buffer_size 32k;
    large_client_header_buffers 4 32k;

    sendfile       on;

    #tcp_nopush     on;
    tcp_nodelay    on;

    client_header_timeout 30;
    client_body_timeout 30;
    send_timeout   30;
    client_max_body_size 100M;

    keepalive_timeout  60;

    proxy_connect_timeout  100;
    proxy_send_timeout  100;
    proxy_read_timeout  100;
    proxy_buffer_size  16k;
    proxy_buffers  4 32k;
    proxy_busy_buffers_size  64k;
    
    gzip  on;
    gzip_min_length 1k;
    gzip_buffers 4 16k;
    gzip_http_version 1.1;
    gzip_comp_level 2;
    gzip_types text/plain application/x-javascript text/css application/xml text/javascript;
    gzip_vary on;

    upstream web1{
      ip_hash;
      server 192.168.247.128:8080 max_fails=3 fail_timeout=30s;
      server 192.168.247.131:8080 max_fails=3 fail_timeout=30s;
    }


    server {
        listen       443;
        server_name  www.scj.com;
        root /opt/nginx/www/www.scj.com;
        index index.html index.htm index.jsp;

        ssl on;
        ssl_certificate /usr/local/nginx/ca/server.crt;
        ssl_certificate_key /usr/local/nginx/ca/server.key;

        access_log  logs/www.scj.com/access.log main;
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        location ~ .*\.jsp$ {
          proxy_pass http://web1;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_next_upstream http_502 http_504 error timeout invalid_header;
        }

        location ~ .*\.(gif|jpg|png|swf|bmp|jpeg)$ {
            expires 10d;
        }

        location ~ .*\.(js|css)?$ {
            expires 1h;
        }
    }

    #server {
       #listen 80 default;
       #server_name _;
       #return 500;
    #}
}
##创建日志目录和根目录
mkdir /usr/local/nginx/logs/www.scj.com        ##日志目录
mkdir -p /opt/nginx/www/www.scj.com            ##创建根目录
##启动nginx:
# /usr/local/nginx/sbin/nginx

# ps -ef | grep nginx
root       7890      1  0 08:00 ?        00:00:00 nginx: master process /usr/local/nginx/sbin/nginx
nginx      7891   7890  0 08:00 ?        00:00:00 nginx: worker process      
nginx      7892   7890  0 08:00 ?        00:00:00 nginx: worker process      
root       7896   7798  0 08:00 pts/0    00:00:00 grep nginx

# netstat  -tlnpa | grep nginx
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      7890/nginx

##设置开机自动启动





    部署keepalived,实现自动切换:

        安装keepalived:(主,备)

cd /usr/local/src/
wget http://www.keepalived.org/software/keepalived-1.2.15.tar.gz
tar -zxf keepalived-1.2.15.tar.gz
cd keepalived-1.2.15
./configure --prefix=/usr/local/keepalived
make
make install

        拷贝文件:(主,备)

cp -a /usr/local/keepalived/etc/rc.d/init.d/keepalived  /etc/init.d/
cp -a /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
mkdir /etc/keepalived/
cp -a /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
cp -a /usr/local/keepalived/sbin/keepalived /usr/sbin/

##注意: /etc/sysconfig/keepalived 和 /etc/keepalived/keepalived.conf 的路径一定要正确,
##因为在执行/etc/init.d/keepalived这个启动脚本时,会读取/etc/sysconfig/keepalived 和 /etc/keepalived/keepalived.conf 这两个文件

        修改配置文件:

            nginx主:

[root@nginxmaster ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.old
[root@nginxmaster ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    notification_email {
      [email protected]
    }
    notification_email_from root@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id nginx_ha
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    #nopreempt

    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.247.150
    }
}

virtual_server 192.168.247.150 443 {
    delay_loop 6
    #lb_algo wrr
    #lb_kind DR
    #persistence_timeout 50
    protocol TCP
    real_server 192.168.247.129 443 {
        #weight 3
        notify_down /etc/keepalived/killkeepalived.sh
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 443
        }
    }
}


           nginx备:

[root@nginxslave ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.old
[root@nginxslave ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    notification_email {
      [email protected]
    }
    notification_email_from root@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id nginx_ha
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    #nopreempt

    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.247.150
    }
}

virtual_server 192.168.247.150 443 {
    delay_loop 6
    #lb_algo wrr
    #lb_kind DR
    #persistence_timeout 50
    protocol TCP
    real_server 192.168.247.130 443 {
        #weight 3
        notify_down /etc/keepalived/killkeepalived.sh
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 443
        }
    }
}

            创建/etc/keepalived/killkeepalived.sh脚本:(主,备)

vi /etc/keepalived/killkeepalived.sh
#!/bin/bash
# check nginx server status

nginx_status=`netstat  -tlnpa | grep 443 | wc -l`

if [ $nginx_status -eq 0 ];then
  /usr/local/nginx/sbin/nginx
  sleep 1
  nginx_status=`netstat  -tlnpa | grep 443 | wc -l`
  if [ $nginx_status -eq 0 ];then
    /etc/init.d/keepalived stop
  fi
fi

chmod 755 /etc/keepalived/killkeepalived.sh


        修改keepalived的日志文件:(主,备)

            参考:http://chenwenming.blog.51cto.com/327092/745316

            说明:

##centos6.3之后的syslog改名叫rsyslog了,默认在 /etc/rsyslog.conf

            修改/etc/sysconfig/keepalived:

vi /etc/sysconfig/keepalived
# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp               -P    Only run with VRRP subsystem.
# --check              -C    Only run with Health-checker subsystem.
# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.
# --dump-conf          -d    Dump the configuration data.
# --log-detail         -D    Detailed log messages.
# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)
#

#KEEPALIVED_OPTIONS="-D"
KEEPALIVED_OPTIONS="-D -d -S 0"               ##在最后添加此行

            修改/etc/rsyslog.conf:

vi /etc/rsyslog.conf
##在最后添加此行:
local0.*                                                /var/log/keepalived.log

## /etc/init.d/rsyslog restart
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]


        启动keepalived:

            nginx主:

[root@nginxmaster ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
##查看日志:
Sep 23 20:16:27 nginxmaster Keepalived[2909]: Starting Keepalived v1.2.15 (09/23,2015)
Sep 23 20:16:27 nginxmaster Keepalived[2910]: Starting Healthcheck child process, pid=2912
Sep 23 20:16:27 nginxmaster Keepalived[2910]: Starting VRRP child process, pid=2913
Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Netlink reflector reports IP 192.168.247.129 added
Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Netlink reflector reports IP fe80::20c:29ff:fee0:d236 added
Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering Kernel netlink reflector
Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering Kernel netlink command channel
Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering gratuitous ARP shared channel
Sep 23 20:16:27 nginxmaster kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Sep 23 20:16:27 nginxmaster kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Sep 23 20:16:27 nginxmaster kernel: IPVS: ipvs loaded.
Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP 192.168.247.129 added
Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP fe80::20c:29ff:fee0:d236 added
Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Registering Kernel netlink reflector
Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Registering Kernel netlink command channel
Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Configuration is using : 63272 Bytes
Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Using LinkWatch kernel netlink reflector...
Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Configuration is using : 11705 Bytes
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: IPVS: Scheduler or persistence engine not found
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: IPVS: No such process
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Using LinkWatch kernel netlink reflector...
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Activating healthchecker for service [192.168.247.129]:443
Sep 23 20:16:28 nginxmaster kernel: IPVS: Scheduler module ip_vs_ not found
Sep 23 20:16:31 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 23 20:16:32 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP 192.168.247.150 added
Sep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150
Sep 23 20:16:37 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150


##设置开机自动启动
chkconfig keepalived on

            nginx备:

[root@nginxslave ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
##查看日志:
Sep 24 09:43:29 nginxslave Keepalived[9481]: Starting Keepalived v1.2.15 (09/24,2015)
Sep 24 09:43:29 nginxslave Keepalived[9482]: Starting Healthcheck child process, pid=9484
Sep 24 09:43:29 nginxslave Keepalived[9482]: Starting VRRP child process, pid=9485
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Netlink reflector reports IP 192.168.247.130 added
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Netlink reflector reports IP fe80::20c:29ff:fe1e:8eab added
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering Kernel netlink reflector
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering Kernel netlink command channel
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering gratuitous ARP shared channel
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Configuration is using : 63262 Bytes
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Using LinkWatch kernel netlink reflector...
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Netlink reflector reports IP 192.168.247.130 added
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Netlink reflector reports IP fe80::20c:29ff:fe1e:8eab added
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Registering Kernel netlink reflector
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Registering Kernel netlink command channel
Sep 24 09:43:29 nginxslave kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Sep 24 09:43:29 nginxslave kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Sep 24 09:43:29 nginxslave kernel: IPVS: ipvs loaded.
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Configuration is using : 11695 Bytes
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: IPVS: Scheduler or persistence engine not found
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: IPVS: No such process
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Using LinkWatch kernel netlink reflector...
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Activating healthchecker for service [192.168.247.130]:443
Sep 24 09:43:29 nginxslave kernel: IPVS: Scheduler module ip_vs_ not found


##设置开机自动启动
chkconfig keepalived on


        查看vip绑定在哪台机器上:

[root@nginxmaster ~]#ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e0:d2:36 brd ff:ff:ff:ff:ff:ff
    inet 192.168.247.129/24 brd 192.168.247.255 scope global eth0
    inet 192.168.247.150/32 scope global eth0
    inet6 fe80::20c:29ff:fee0:d236/64 scope link 
       valid_lft forever preferred_lft forever
[root@nginxslave ~]#ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:1e:8e:ab brd ff:ff:ff:ff:ff:ff
    inet 192.168.247.130/24 brd 192.168.247.255 scope global eth0
    inet6 fe80::20c:29ff:fe1e:8eab/64 scope link 
       valid_lft forever preferred_lft forever
##由上发现vip150绑定在nginxmaster,正常





测试:

    测试一:把nginx主的nginx服务stop掉

[root@nginxmaster ~]# /usr/local/nginx/sbin/nginx -s stop
##查看日志:
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: TCP connection to [192.168.247.129]:443 failed !!!
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not defined
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Executing [/etc/keepalived/killkeepalived.sh] for service [192.168.247.129]:443 in VS [192.168.247.150]:443
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Lost quorum 1-0=1 > 0 for VS [192.168.247.150]:443
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Remote SMTP server [127.0.0.1]:25 connected.
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: SMTP alert successfully sent.
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: TCP connection to [192.168.247.129]:443 success.
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Adding service [192.168.247.129]:443 to VS [192.168.247.150]:443
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not defined
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Gained quorum 1+0=1 <= 1 for VS [192.168.247.150]:443
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Remote SMTP server [127.0.0.1]:25 connected.
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: SMTP alert successfully sent

##keepalived会先把129移除,然后执行/etc/keepalived/killkeepalived.sh脚本,nginx服务正常启动,又把129加入到集群中

    测试二:把nginx主的keepalived服务stop掉

[root@nginxmaster ~]# /etc/init.d/keepalived stop
Stopping keepalived:                                       [  OK  ]
##查看nginx主的日志:
Sep 23 23:41:05 nginxmaster Keepalived[3988]: Stopping Keepalived v1.2.15 (09/23,2015)
Sep 23 23:41:05 nginxmaster Keepalived_vrrp[3991]: VRRP_Instance(VI_1) sending 0 priority
Sep 23 23:41:05 nginxmaster Keepalived_vrrp[3991]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: Netlink reflector reports IP 192.168.247.150 removed
Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443
Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not defined
Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: IPVS: No such service
##查看nginx备日志:
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: TCP connection to [192.168.247.130]:443 success.
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Adding service [192.168.247.130]:443 to VS [192.168.247.150]:443
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: IPVS: Service not defined
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Gained quorum 1+0=1 <= 1 for VS [192.168.247.150]:443
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Remote SMTP server [127.0.0.1]:25 connected.
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: SMTP alert successfully sent.
Sep 24 12:58:05 nginxslave dhclient[906]: DHCPREQUEST on eth0 to 192.168.247.254 port 67 (xid=0x10a73226)
Sep 24 12:58:05 nginxslave dhclient[906]: DHCPACK from 192.168.247.254 (xid=0x10a73226)
Sep 24 12:58:07 nginxslave dhclient[906]: bound to 192.168.247.130 -- renewal in 705 seconds.
Sep 24 13:05:23 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 24 13:05:24 nginxslave Keepalived_healthcheckers[10012]: Netlink reflector reports IP 192.168.247.150 added
Sep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150
Sep 24 13:05:29 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150

##nginx备变成master,绑定vip 150

    测试三:把nginx主的nginx服务stop,并不再让它成功启动

##删掉日志目录,nginx便不能启动了
[root@nginxmaster logs]# rm -rf /usr/local/nginx/logs/www.scj.com/
##关闭nginx服务
[root@nginxmaster ~]# /usr/local/nginx/sbin/nginx -s stop
##查看nginx主日志:
Sep 23 23:44:33 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: TCP connection to [192.168.247.129]:443 failed !!!
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: IPVS: Service not defined
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Executing [/etc/keepalived/killkeepalived.sh] for service [192.168.247.129]:443 in VS [192.168.247.150]:443
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Lost quorum 1-0=1 > 0 for VS [192.168.247.150]:443
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Remote SMTP server [127.0.0.1]:25 connected.
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: SMTP alert successfully sent.
Sep 23 23:44:39 nginxmaster Keepalived[4076]: Stopping Keepalived v1.2.15 (09/23,2015)
Sep 23 23:44:39 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) sending 0 priority
Sep 23 23:44:39 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 23 23:44:39 nginxmaster Keepalived_healthcheckers[4078]: Netlink reflector reports IP 192.168.247.150 removed
Sep 23 23:44:39 nginxmaster Keepalived_healthcheckers[4078]: IPVS: No such service

##nginx服务不能启动了,然后把keepalived服务关掉
##查看nginx备日志
Sep 24 13:08:57 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150
Sep 24 13:08:58 nginxslave Keepalived_healthcheckers[10012]: Netlink reflector reports IP 192.168.247.150 added
Sep 24 13:09:03 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150
Sep 24 13:09:52 nginxslave dhclient[906]: DHCPREQUEST on eth0 to 192.168.247.254 port 67 (xid=0x10a73226)
Sep 24 13:09:52 nginxslave dhclient[906]: DHCPACK from 192.168.247.254 (xid=0x10a73226)
Sep 24 13:09:54 nginxslave dhclient[906]: bound to 192.168.247.130 -- renewal in 800 seconds.

##nginx备变为master,并绑定vip 150

    测试四:把nginx备的nginx服务stop

##查看nginx备的nginx服务会不会又正常启动

    测试五:把nginx备的keepalived服务stop

##nginx主不会有任何变化

    测试六:把nginx备的nginx服务stop掉,并不再让它正常启动

##nginx备的keepalived服务会被关掉
##nginx主不会有任何变化


注意:

    修复后,一定要记得把keepalived服务启动起来








nfs部署:

    http://732233048.blog.51cto.com/9323668/1669417