nginx+keepalived+tomcat配置高可用web集群

nginx+keepalived+tomcat配置高可用web集群

基本架构：

    

角色	ip	安装软件	作用	主机名
nginx主	192.168.247.129	nginx+keepalived	反向代理	nginxmaster.com
nginx备	192.168.247.130	nginx+keepalived	反向代理	nginxsalve.com
tomcat1	192.168.247.128	tomcat	web服务器	tomcat1.com
tomcat2	192.168.247.131	tomcat	web服务器	tomcat2.com
nfs主	192.168.247.132	nfs	低端共享存储	nfsmaster.com
nfs备	192.168.247.133	nfs	低端共享存储	nfsslave.com
虚拟ip	192.168.247.150

集群部署：

    tomcat安装部署：

        编译安装tomcat：（tomcat1，tomcat2）

            安装JDK，运行java环境：（tomcat1，tomcat2）

                卸载centos自带的JDK：

rpm -qa | grep java
yum -y remove java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5

                下载安装JDK：

##下载地址：http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html 
##下载最新的（64位）：jdk-7u79-linux-x64.rpm

##在/usr下建立一个java目录，以备将java程序安装在此目录下
mkdir /usr/java
cd /usr/local/src
rpm -ivh jdk-7u79-linux-x64.rpm        ##安装后，在/usr/java目录下就会生成一个新的目录jdk1.7.0_79，该目录下存放的是安装后的文件

##设置环境变量：
vi /etc/profile  ##在最后添加：
export JAVA_HOME=/usr/java/jdk1.7.0_79
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

source /etc/profile

##验证是否安装成功：
java -version         ##出现如下则正常
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

            安装tomcat7：（tomcat1，tomcat2）

软件：apache-tomcat-7.0.29.tar.gz
cd /usr/local/src/
tar -zxf apache-tomcat-7.0.42.tar.gz
cp -a apache-tomcat-7.0.42 /usr/local/tomcat7/

/usr/local/tomcat7/bin/startup.sh       ##启动tomcat，出现如下则正常
Using CATALINA_BASE:   /usr/local/tomcat7
Using CATALINA_HOME:   /usr/local/tomcat7
Using CATALINA_TMPDIR: /usr/local/tomcat7/temp
Using JRE_HOME:        /usr/java/jdk1.7.0_79
Using CLASSPATH:       /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar

##设置开机自动启动

##访问下：    ip:8080   看是否正常

        配置tomcat：（tomcat1，tomcat2）

            配置server.xml文件：

cd /usr/local/tomcat7/conf/
vi server.xml
##找到：
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
##将其注释掉

##添加如下（已优化）：
    <Connector  port="8080"   
  
                protocol="HTTP/1.1"  
  
                maxHttpHeaderSize="8192"  
  
                maxThreads="1000"  
  
                minSpareThreads="100"  
  
                maxSpareThreads="1000"  
  
                minProcessors="100"  
  
                maxProcessors="1000"  
  
                enableLookups="false"  
  
                compression="on"  
  
                compressionMinSize="2048"  
  
                compressableMimeType="text/html,text/xml,text/javascript,text/css,text/plain"  
  
                connectionTimeout="20000"  
  
                URIEncoding="utf-8"  
  
                acceptCount="1000"  
  
                redirectPort="8443"  
  
                disableUploadTimeout="true"/>
##配置优化： 
            优化JVM堆内存：
                vi /usr/local/tomcat7/bin/catalina.sh
                添加：JAVA_OPTS='-server -Xms1024m -Xmx2048m -XX:PermSize=256M -XX:MaxNewSize=256m -XX:MaxPermSize=256m'
		
	    修改日志目录：/data/tomcat7/logs
	        vi /usr/local/tomcat7/conf/server.xml
			<Valve className="org.apache.catalina.valves.AccessLogValve" directory="/data/tomcat7/logs"
			
		vi /usr/local/tomcat7/conf/logging.properties
			1catalina.org.apache.juli.FileHandler.level = FINE
			1catalina.org.apache.juli.FileHandler.directory = /data/tomcat7/logs
			1catalina.org.apache.juli.FileHandler.prefix = catalina.

			2localhost.org.apache.juli.FileHandler.level = FINE
			2localhost.org.apache.juli.FileHandler.directory = /data/tomcat7/logs
			2localhost.org.apache.juli.FileHandler.prefix = localhost.

			3manager.org.apache.juli.FileHandler.level = FINE
			3manager.org.apache.juli.FileHandler.directory = /data/tomcat7/logs
			3manager.org.apache.juli.FileHandler.prefix = manager.

			4host-manager.org.apache.juli.FileHandler.level = FINE
			4host-manager.org.apache.juli.FileHandler.directory = /data/tomcat7/logs
			4host-manager.org.apache.juli.FileHandler.prefix = host-manager.
		
		vi /usr/local/tomcat7/bin/catalina.sh
			if [ -z "$CATALINA_OUT" ] ; then
				CATALINA_OUT=/data/tomcat7/logs/catalina.out
			fi
	
	    修改日志输出格式：
	        vi /usr/local/tomcat7/conf/server.xml
			pattern="%{X-Real-IP}i %l %u %t &quot;%r&quot; %s %b" />            #nginx做反向代理，%{X-Real-IP}i 获取用户的真实ip              

                
                    ##找到：Valve标签
                    ##将pattern参数的值修改为："%{X-Real-IP}i %l %u %t &quot;%r&quot; %s %b"        ##{X-Real-IP}i，获取的是真实客户端ip，不是代理机ip
                    ##每个参数的含义：http://twb.iteye.com/blog/182100
                    ##nginx前端代理tomcat取真实客户端IP：http://www.cnphp6.com/archives/60908

##重新启动tomcat：
[root@tomcat1 ~]# /usr/local/tomcat7/bin/shutdown.sh 
Using CATALINA_BASE:   /usr/local/tomcat7
Using CATALINA_HOME:   /usr/local/tomcat7
Using CATALINA_TMPDIR: /usr/local/tomcat7/temp
Using JRE_HOME:        /usr/java/jdk1.7.0_79
Using CLASSPATH:       /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar
[root@tomcat1 ~]# /usr/local/tomcat7/bin/startup.sh 
Using CATALINA_BASE:   /usr/local/tomcat7
Using CATALINA_HOME:   /usr/local/tomcat7
Using CATALINA_TMPDIR: /usr/local/tomcat7/temp
Using JRE_HOME:        /usr/java/jdk1.7.0_79
Using CLASSPATH:       /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar

##tomcat性能优化：
参考：http://www.cnblogs.com/ggjucheng/archive/2013/04/16/3024731.html

    反向代理的部署：

        编译安装nginx：（主，备）

            安装nginx的依赖包：

yum -y install gcc gcc-c++ autoconf automake
yum -y install zlib zlib-devel openssl openssl-devel pcre-devel

            编译安装pcre，使nginx支持rewrite重写：

cd /usr/local/src/
wget http://sourceforge.net/projects/pcre/files/pcre/8.35/pcre-8.35.tar.gz
tar -zxf pcre-8.35.tar.gz
cd pcre-8.35
./configure
make
make install

            安装nginx：

cd /usr/local/src/
wget http://nginx.org/download/nginx-1.6.2.tar.gz
tar -zxf nginx-1.6.2.tar.gz 
cd nginx-1.6.2
./configure  --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_gzip_static_module --with-http_flv_module --with-http_gunzip_module --with-http_realip_module
make
make install
###注意--with-http_ssl_module，后面要用ssl

        配置nginx：（主，备）

            创建nginx用户：

useradd -s /sbin/nologin nginx

            修改文件打开数：

ulimit -n 65535
##设置开机自动启动

            生成ssl证书（模拟）（忽略）

##创建证书存放目录：
cd /usr/local/nginx/
mkdir ca
cd ca
##创建服务器私钥：
openssl genrsa -des3 -out server.key 1024
##创建签名请求的证书（CSR）：
openssl req -new -key server.key -out server.csr
##在加载SSL支持的Nginx并使用上述私钥时除去必须的口令
cp server.key server.key.org
openssl rsa -in server.key.org -out server.key
##最后标记证书使用上述私钥和CSR：
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

         生成ssl证书（模拟）

##创建证书存放目录：
mkdir /usr/local/nginx/ca
cd /usr/local/nginx/ca
##生成密钥文件key和csr文件：
openssl req -new -newkey rsa:2048 -nodes -keyout server.key -out server.csr
##生成服务器认证文件crt：
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

           修改配置文件：

# mv /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf.old
# vi /usr/local/nginx/conf/nginx.conf
user nginx nginx;
worker_processes  2;
error_log  logs/error.log;
pid        logs/nginx.pid;
worker_rlimit_nofile 65535;

events {
    use epoll;
    worker_connections  65535;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" $http_x_forwarded_for';
    
    server_names_hash_bucket_size 128;
    client_header_buffer_size 32k;
    large_client_header_buffers 4 32k;

    sendfile       on;

    #tcp_nopush     on;
    tcp_nodelay    on;

    client_header_timeout 30;
    client_body_timeout 30;
    send_timeout   30;
    client_max_body_size 100M;

    keepalive_timeout  60;

    proxy_connect_timeout  100;
    proxy_send_timeout  100;
    proxy_read_timeout  100;
    proxy_buffer_size  16k;
    proxy_buffers  4 32k;
    proxy_busy_buffers_size  64k;
    
    gzip  on;
    gzip_min_length 1k;
    gzip_buffers 4 16k;
    gzip_http_version 1.1;
    gzip_comp_level 2;
    gzip_types text/plain application/x-javascript text/css application/xml text/javascript;
    gzip_vary on;

    upstream web1{
      ip_hash;
      server 192.168.247.128:8080 max_fails=3 fail_timeout=30s;
      server 192.168.247.131:8080 max_fails=3 fail_timeout=30s;
    }


    server {
        listen       443;
        server_name  www.scj.com;
        root /opt/nginx/www/www.scj.com;
        index index.html index.htm index.jsp;

        ssl on;
        ssl_certificate /usr/local/nginx/ca/server.crt;
        ssl_certificate_key /usr/local/nginx/ca/server.key;

        access_log  logs/www.scj.com/access.log main;
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        location ~ .*\.jsp$ {
          proxy_pass http://web1;
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_next_upstream http_502 http_504 error timeout invalid_header;
        }

        location ~ .*\.(gif|jpg|png|swf|bmp|jpeg)$ {
            expires 10d;
        }

        location ~ .*\.(js|css)?$ {
            expires 1h;
        }
    }

    #server {
       #listen 80 default;
       #server_name _;
       #return 500;
    #}
}

##创建日志目录和根目录
mkdir /usr/local/nginx/logs/www.scj.com        ##日志目录
mkdir -p /opt/nginx/www/www.scj.com            ##创建根目录

##启动nginx：
# /usr/local/nginx/sbin/nginx

# ps -ef | grep nginx
root       7890      1  0 08:00 ?        00:00:00 nginx: master process /usr/local/nginx/sbin/nginx
nginx      7891   7890  0 08:00 ?        00:00:00 nginx: worker process      
nginx      7892   7890  0 08:00 ?        00:00:00 nginx: worker process      
root       7896   7798  0 08:00 pts/0    00:00:00 grep nginx

# netstat  -tlnpa | grep nginx
tcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      7890/nginx

##设置开机自动启动

    部署keepalived，实现自动切换：

        安装keepalived：（主，备）

cd /usr/local/src/
wget http://www.keepalived.org/software/keepalived-1.2.15.tar.gz
tar -zxf keepalived-1.2.15.tar.gz
cd keepalived-1.2.15
./configure --prefix=/usr/local/keepalived
make
make install

        拷贝文件：（主，备）

cp -a /usr/local/keepalived/etc/rc.d/init.d/keepalived  /etc/init.d/
cp -a /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
mkdir /etc/keepalived/
cp -a /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
cp -a /usr/local/keepalived/sbin/keepalived /usr/sbin/

##注意： /etc/sysconfig/keepalived 和 /etc/keepalived/keepalived.conf 的路径一定要正确，
##因为在执行/etc/init.d/keepalived这个启动脚本时，会读取/etc/sysconfig/keepalived 和 /etc/keepalived/keepalived.conf 这两个文件

        修改配置文件：

            nginx主：

[root@nginxmaster ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.old
[root@nginxmaster ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    notification_email {
      [email protected]
    }
    notification_email_from root@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id nginx_ha
}

vrrp_instance VI_1 {
    state master
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    #nopreempt

    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.247.150
    }
}

virtual_server 192.168.247.150 443 {
    delay_loop 6
    #lb_algo wrr
    #lb_kind DR
    #persistence_timeout 50
    protocol TCP
    real_server 192.168.247.129 443 {
        #weight 3
        notify_down /etc/keepalived/killkeepalived.sh
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 443
        }
    }
}

           nginx备：

[root@nginxslave ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.old
[root@nginxslave ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    notification_email {
      [email protected]
    }
    notification_email_from root@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id nginx_ha
}

vrrp_instance VI_1 {
    state backup
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    #nopreempt

    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.247.150
    }
}

virtual_server 192.168.247.150 443 {
    delay_loop 6
    #lb_algo wrr
    #lb_kind DR
    #persistence_timeout 50
    protocol TCP
    real_server 192.168.247.130 443 {
        #weight 3
        notify_down /etc/keepalived/killkeepalived.sh
        TCP_CHECK {
            connect_timeout 10
            nb_get_retry 3
            delay_before_retry 3
            connect_port 443
        }
    }
}

            创建/etc/keepalived/killkeepalived.sh脚本：（主，备）

vi /etc/keepalived/killkeepalived.sh
#!/bin/bash
# check nginx server status

nginx_status=`netstat  -tlnpa | grep 443 | wc -l`

if [ $nginx_status -eq 0 ];then
  /usr/local/nginx/sbin/nginx
  sleep 1
  nginx_status=`netstat  -tlnpa | grep 443 | wc -l`
  if [ $nginx_status -eq 0 ];then
    /etc/init.d/keepalived stop
  fi
fi

chmod 755 /etc/keepalived/killkeepalived.sh

        修改keepalived的日志文件：（主，备）

            参考：http://chenwenming.blog.51cto.com/327092/745316

            说明：

##centos6.3之后的syslog改名叫rsyslog了，默认在 /etc/rsyslog.conf

            修改/etc/sysconfig/keepalived：

vi /etc/sysconfig/keepalived
# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp               -P    Only run with VRRP subsystem.
# --check              -C    Only run with Health-checker subsystem.
# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.
# --dump-conf          -d    Dump the configuration data.
# --log-detail         -D    Detailed log messages.
# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)
#

#KEEPALIVED_OPTIONS="-D"
KEEPALIVED_OPTIONS="-D -d -S 0"               ##在最后添加此行

            修改/etc/rsyslog.conf:

vi /etc/rsyslog.conf
##在最后添加此行：
local0.*                                                /var/log/keepalived.log

## /etc/init.d/rsyslog restart
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]

        启动keepalived：

            nginx主：

[root@nginxmaster ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]

##查看日志：
Sep 23 20:16:27 nginxmaster Keepalived[2909]: Starting Keepalived v1.2.15 (09/23,2015)
Sep 23 20:16:27 nginxmaster Keepalived[2910]: Starting Healthcheck child process, pid=2912
Sep 23 20:16:27 nginxmaster Keepalived[2910]: Starting VRRP child process, pid=2913
Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Netlink reflector reports IP 192.168.247.129 added
Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Netlink reflector reports IP fe80::20c:29ff:fee0:d236 added
Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering Kernel netlink reflector
Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering Kernel netlink command channel
Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering gratuitous ARP shared channel
Sep 23 20:16:27 nginxmaster kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Sep 23 20:16:27 nginxmaster kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Sep 23 20:16:27 nginxmaster kernel: IPVS: ipvs loaded.
Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP 192.168.247.129 added
Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP fe80::20c:29ff:fee0:d236 added
Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Registering Kernel netlink reflector
Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Registering Kernel netlink command channel
Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Configuration is using : 63272 Bytes
Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Using LinkWatch kernel netlink reflector...
Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Configuration is using : 11705 Bytes
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: IPVS: Scheduler or persistence engine not found
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: IPVS: No such process
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Using LinkWatch kernel netlink reflector...
Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Activating healthchecker for service [192.168.247.129]:443
Sep 23 20:16:28 nginxmaster kernel: IPVS: Scheduler module ip_vs_ not found
Sep 23 20:16:31 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 23 20:16:32 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP 192.168.247.150 added
Sep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150
Sep 23 20:16:37 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150


##设置开机自动启动
chkconfig keepalived on

            nginx备：

[root@nginxslave ~]# /etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]

##查看日志：
Sep 24 09:43:29 nginxslave Keepalived[9481]: Starting Keepalived v1.2.15 (09/24,2015)
Sep 24 09:43:29 nginxslave Keepalived[9482]: Starting Healthcheck child process, pid=9484
Sep 24 09:43:29 nginxslave Keepalived[9482]: Starting VRRP child process, pid=9485
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Netlink reflector reports IP 192.168.247.130 added
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Netlink reflector reports IP fe80::20c:29ff:fe1e:8eab added
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering Kernel netlink reflector
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering Kernel netlink command channel
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering gratuitous ARP shared channel
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Configuration is using : 63262 Bytes
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Using LinkWatch kernel netlink reflector...
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Netlink reflector reports IP 192.168.247.130 added
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Netlink reflector reports IP fe80::20c:29ff:fe1e:8eab added
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Registering Kernel netlink reflector
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Registering Kernel netlink command channel
Sep 24 09:43:29 nginxslave kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Sep 24 09:43:29 nginxslave kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Sep 24 09:43:29 nginxslave kernel: IPVS: ipvs loaded.
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Configuration is using : 11695 Bytes
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: IPVS: Scheduler or persistence engine not found
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: IPVS: No such process
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Using LinkWatch kernel netlink reflector...
Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Activating healthchecker for service [192.168.247.130]:443
Sep 24 09:43:29 nginxslave kernel: IPVS: Scheduler module ip_vs_ not found


##设置开机自动启动
chkconfig keepalived on

        查看vip绑定在哪台机器上：

[root@nginxmaster ~]#ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:e0:d2:36 brd ff:ff:ff:ff:ff:ff
    inet 192.168.247.129/24 brd 192.168.247.255 scope global eth0
    inet 192.168.247.150/32 scope global eth0
    inet6 fe80::20c:29ff:fee0:d236/64 scope link 
       valid_lft forever preferred_lft forever

[root@nginxslave ~]#ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:1e:8e:ab brd ff:ff:ff:ff:ff:ff
    inet 192.168.247.130/24 brd 192.168.247.255 scope global eth0
    inet6 fe80::20c:29ff:fe1e:8eab/64 scope link 
       valid_lft forever preferred_lft forever

##由上发现vip150绑定在nginxmaster，正常

测试：

    测试一：把nginx主的nginx服务stop掉

[root@nginxmaster ~]# /usr/local/nginx/sbin/nginx -s stop

##查看日志：
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: TCP connection to [192.168.247.129]:443 failed !!!
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not defined
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Executing [/etc/keepalived/killkeepalived.sh] for service [192.168.247.129]:443 in VS [192.168.247.150]:443
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Lost quorum 1-0=1 > 0 for VS [192.168.247.150]:443
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Remote SMTP server [127.0.0.1]:25 connected.
Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: SMTP alert successfully sent.
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: TCP connection to [192.168.247.129]:443 success.
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Adding service [192.168.247.129]:443 to VS [192.168.247.150]:443
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not defined
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Gained quorum 1+0=1 <= 1 for VS [192.168.247.150]:443
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Remote SMTP server [127.0.0.1]:25 connected.
Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: SMTP alert successfully sent

##keepalived会先把129移除，然后执行/etc/keepalived/killkeepalived.sh脚本，nginx服务正常启动，又把129加入到集群中

    测试二：把nginx主的keepalived服务stop掉

[root@nginxmaster ~]# /etc/init.d/keepalived stop
Stopping keepalived:                                       [  OK  ]

##查看nginx主的日志：
Sep 23 23:41:05 nginxmaster Keepalived[3988]: Stopping Keepalived v1.2.15 (09/23,2015)
Sep 23 23:41:05 nginxmaster Keepalived_vrrp[3991]: VRRP_Instance(VI_1) sending 0 priority
Sep 23 23:41:05 nginxmaster Keepalived_vrrp[3991]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: Netlink reflector reports IP 192.168.247.150 removed
Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443
Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not defined
Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: IPVS: No such service

##查看nginx备日志：
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: TCP connection to [192.168.247.130]:443 success.
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Adding service [192.168.247.130]:443 to VS [192.168.247.150]:443
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: IPVS: Service not defined
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Gained quorum 1+0=1 <= 1 for VS [192.168.247.150]:443
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Remote SMTP server [127.0.0.1]:25 connected.
Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: SMTP alert successfully sent.
Sep 24 12:58:05 nginxslave dhclient[906]: DHCPREQUEST on eth0 to 192.168.247.254 port 67 (xid=0x10a73226)
Sep 24 12:58:05 nginxslave dhclient[906]: DHCPACK from 192.168.247.254 (xid=0x10a73226)
Sep 24 12:58:07 nginxslave dhclient[906]: bound to 192.168.247.130 -- renewal in 705 seconds.
Sep 24 13:05:23 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 24 13:05:24 nginxslave Keepalived_healthcheckers[10012]: Netlink reflector reports IP 192.168.247.150 added
Sep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150
Sep 24 13:05:29 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150

##nginx备变成master，绑定vip 150

    测试三：把nginx主的nginx服务stop，并不再让它成功启动

##删掉日志目录，nginx便不能启动了
[root@nginxmaster logs]# rm -rf /usr/local/nginx/logs/www.scj.com/

##关闭nginx服务
[root@nginxmaster ~]# /usr/local/nginx/sbin/nginx -s stop

##查看nginx主日志：
Sep 23 23:44:33 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: TCP connection to [192.168.247.129]:443 failed !!!
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: IPVS: Service not defined
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Executing [/etc/keepalived/killkeepalived.sh] for service [192.168.247.129]:443 in VS [192.168.247.150]:443
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Lost quorum 1-0=1 > 0 for VS [192.168.247.150]:443
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Remote SMTP server [127.0.0.1]:25 connected.
Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: SMTP alert successfully sent.
Sep 23 23:44:39 nginxmaster Keepalived[4076]: Stopping Keepalived v1.2.15 (09/23,2015)
Sep 23 23:44:39 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) sending 0 priority
Sep 23 23:44:39 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 23 23:44:39 nginxmaster Keepalived_healthcheckers[4078]: Netlink reflector reports IP 192.168.247.150 removed
Sep 23 23:44:39 nginxmaster Keepalived_healthcheckers[4078]: IPVS: No such service

##nginx服务不能启动了，然后把keepalived服务关掉

##查看nginx备日志
Sep 24 13:08:57 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150
Sep 24 13:08:58 nginxslave Keepalived_healthcheckers[10012]: Netlink reflector reports IP 192.168.247.150 added
Sep 24 13:09:03 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150
Sep 24 13:09:52 nginxslave dhclient[906]: DHCPREQUEST on eth0 to 192.168.247.254 port 67 (xid=0x10a73226)
Sep 24 13:09:52 nginxslave dhclient[906]: DHCPACK from 192.168.247.254 (xid=0x10a73226)
Sep 24 13:09:54 nginxslave dhclient[906]: bound to 192.168.247.130 -- renewal in 800 seconds.

##nginx备变为master，并绑定vip 150

    测试四：把nginx备的nginx服务stop

##查看nginx备的nginx服务会不会又正常启动

    测试五：把nginx备的keepalived服务stop

##nginx主不会有任何变化

    测试六：把nginx备的nginx服务stop掉，并不再让它正常启动

##nginx备的keepalived服务会被关掉
##nginx主不会有任何变化

注意：

    修复后，一定要记得把keepalived服务启动起来

nfs部署：

    http://732233048.blog.51cto.com/9323668/1669417