nginx1.2.8+keepalived1.2.24实现高可用负载均衡

标签（空格分隔）： nginx keepalived

目录.png

操作环境

两台Linux虚拟机：Red Hat Enterprise Linux Server release 6.5(Santiago)

[root@localhost logs]# uname -a
Linux localhost 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost logs]# cat /etc/redhat-release 
CentOS release 6.5 (Final)

背景

每个机器的keepalived执行脚本监控nginx是否执行，当发现本机的nginx挂掉，就会将vip飘到另一台机器，从而实现高可用，注意：主备机器只有一台机器上面有vip

两台虚拟机的ip地址分别为：10.0.0.78（主），10.10.10.102（备）
keepalived配置了vip：10.0.0.79 供外使用
nginx实现负载，实现代理的地址为：10.0.1.227:9200，10.0.1.45:9200

安装配置

Linux中Tree插件安装

1.下载目录 http://mama.indstate.edu/users/ice/tree/

2.解压安装

tar -zxvf tree-1.7.0.tgz
cd tree-1.7.0
make install
#使用tree进行测试
tree

Nginx安装

除了有一些细微的变动，基本上都是参考这个博客 http://blog.csdn.net/yabingshi_tech/article/details/47416787

1.选定源码目录

选定目录为： /usr/local

nginx1.2.8+keepalived1.2.24实现高可用负载均衡_第2张图片

/usr/local.png

2.安装PCRE库

[root@localhost local]# wget https://sourceforge.net/projects/pcre/files/pcre/8.40/pcre-8.40.tar.gz --no-check-certificate
--2017-11-11 11:32:34--  https://sourceforge.net/projects/pcre/files/pcre/8.40/pcre-8.40.tar.gz
正在解析主机 sourceforge.net... 216.34.181.60
正在连接 sourceforge.net|216.34.181.60|:443... 已连接。
警告: 证书通用名 “*.sourceforge.net” 与所要求的主机名 “sourceforge.net” 不符。
已发出 HTTP 请求，正在等待回应... 302 Found
位置：https://sourceforge.net/projects/pcre/files/pcre/8.40/pcre-8.40.tar.gz/download [跟随至新的 URL]
--2017-11-11 11:32:35--  https://sourceforge.net/projects/pcre/files/pcre/8.40/pcre-8.40.tar.gz/download
正在连接 sourceforge.net|216.34.181.60|:443... 已连接。
警告: 证书通用名 “*.sourceforge.net” 与所要求的主机名 “sourceforge.net” 不符。
已发出 HTTP 请求，正在等待回应... 302 Found
位置：https://downloads.sourceforge.net/project/pcre/pcre/8.40/pcre-8.40.tar.gz?r=&ts=1510371162&use_mirror=jaist [跟随至新的 URL]
--2017-11-11 11:32:36--  https://downloads.sourceforge.net/project/pcre/pcre/8.40/pcre-8.40.tar.gz?r=&ts=1510371162&use_mirror=jaist
正在解析主机 downloads.sourceforge.net... 216.34.181.59
正在连接 downloads.sourceforge.net|216.34.181.59|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 302 Found
位置：https://jaist.dl.sourceforge.net/project/pcre/pcre/8.40/pcre-8.40.tar.gz [跟随至新的 URL]
--2017-11-11 11:32:38--  https://jaist.dl.sourceforge.net/project/pcre/pcre/8.40/pcre-8.40.tar.gz
正在解析主机 jaist.dl.sourceforge.net... 150.65.7.130, 2001:df0:2ed:feed::feed
正在连接 jaist.dl.sourceforge.net|150.65.7.130|:443... 已连接。
警告: 证书通用名 “ftp.jaist.ac.jp” 与所要求的主机名 “jaist.dl.sourceforge.net” 不符。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：2065161 (2.0M) [application/x-gzip]
正在保存至: “pcre-8.40.tar.gz”

76% [===============================================================================================================================================================================>                                                     ] 1,589,248    515K/s eta(英国中部时83% [=============================================================================================================================================================================================>                                       ] 1,720,320    523K/s eta(英国中部时90% [==============================================================================================================================================================================================================>                      ] 1,867,776    532K/s eta(英国中部时96% [============================================================================================================================================================================================================================>        ] 1,998,848    542K/s eta(英国中部时98% [================================================================================================================================================================================================================================>    ] 2,031,616    510K/s eta(英国中部时99% [===================================================================================================================================================================================================================================> ] 2,064,384    486K/s eta(英国中部时100%[====================================================================================================================================================================================================================================>] 2,065,161    486K/s   in 4.1s    

2017-11-11 11:32:42 (492 KB/s) - 已保存 “pcre-8.40.tar.gz” [2065161/2065161])


接着解压安装
tar -xvf pcre-8.36.tar.gz    
cd pcre-8.36  
./configure --enable-utf8  
make  
make install

3.安装zlib库

cd /usr/local/    
[root@ser6-70 local]# wget http://zlib.net/zlib-1.2.8.tar.gz  
--2015-08-10 18:19:44--  http://zlib.net/zlib-1.2.8.tar.gz  
Resolving zlib.net... 69.73.132.10  
Connecting to zlib.net|69.73.132.10|:80... connected.  
HTTP request sent, awaiting response... 200 OK  
Length: 571091 (558K) [application/x-gzip]  
Saving to: “zlib-1.2.8.tar.gz”  
   
100%[===================================================================================================================>] 571,091     12.8K/s   in 25s       
   
2015-08-10 18:20:11 (22.1 KB/s) - “zlib-1.2.8.tar.gz” saved [571091/571091]  

接着解压安装   
tar -zxvf zlib-1.2.8.tar.gz cd zlib-1.2.8  
cd zlib-1.2.8  
./configure  
make  
make install

4.安装ssl

cd /usr/local/  
[root@ser6-70 ~]# wget http://www.openssl.org/source/openssl-1.0.1c.tar.gz  
--2015-08-11 09:25:50--  http://www.openssl.org/source/openssl-1.0.1c.tar.gz  
Resolving www.openssl.org... 194.97.150.234, 2001:608:c00:180::1:ea  
Connecting to www.openssl.org|194.97.150.234|:80... connected.  
HTTP request sent, awaiting response... 200 OK  
Length: 4457113 (4.2M) [application/x-gzip]  
Saving to: “openssl-1.0.1c.tar.gz”  
   
 5% [=====>                                                                                                              ] 247,230     8.09K/s  eta 7m 33s    
 5% [=====>                                                                                                              ] 251,574     7.33K/s  eta 7m 33s    
 5% [=====>                                                                                                              ] 258,814     8.50K/s  eta 7m 32s    
   
   
   
 5% [=====>                                                                                                              ] 258,814     7.85K/s  eta 7m 32s    
 5% [=====>                                                                                                              ] 266,054     8.30K/s  eta 7m 45s    
 6% [======>                                                                                                             ] 283,430     8.95K/s  eta 7m 42s    
   
100%[===================================================================================================================>] 4,457,113   5.58K/s   in 10m 44s   
   
2015-08-11 09:36:35 (6.76 KB/s) - “openssl-1.0.1c.tar.gz” saved [4457113/4457113]  


接着解压安装   
tar -zxvf openssl-1.0.1c.tar.gz  
cd openssl-1.0.1c  
./config  
make  
make install

5.安装nginx

cd /usr/local/
[root@ser6-70 ~]# wget http://nginx.org/download/nginx-1.2.8.tar.gz  
--2015-08-11 09:54:31--  http://nginx.org/download/nginx-1.2.8.tar.gz  
Resolving nginx.org... 206.251.255.63, 2606:7100:1:69::3f  
Connecting to nginx.org|206.251.255.63|:80... connected.  
HTTP request sent, awaiting response... 200 OK  
Length: 730589 (713K) [application/octet-stream]  
Saving to: “nginx-1.2.8.tar.gz”  
   
100%[===================================================================================================================>] 730,589     23.7K/s   in 39s       
   
2015-08-11 09:55:11 (18.4 KB/s) - “nginx-1.2.8.tar.gz” saved [730589/730589]  
tar -zxvf nginx-1.2.8.tar.gz  
cd nginx-1.2.8    
./configure --prefix=/usr/local/nginx   
make  
make install

6.启动nginx

确保系统的 80 端口没被其他程序占用，/usr/local/nginx/sbin/nginx启动nginx

查看占用80端口的进程，kill掉
[root@ser6-70 nginx-1.2.8]# lsof -i:80


若出现
[root@ser6-70 nginx-1.2.8]# lsof -i:80
-bash: lsof: command not found 
通过yum来安装
yum install lsof

若出现这个错误：
[root@ser6-70 nginx-1.2.8]# /usr/local/nginx/sbin/nginx  
/usr/local/nginx/sbin/nginx: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory  

从错误提示信息可以得知是因为缺少lib文件导致，进一步查看具体内容：
[root@ser6-70 nginx-1.2.8]# ldd $(which /usr/local/nginx/sbin/nginx)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00000030e8400000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00000030f9a00000)
libpcre.so.1 => not found
libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00000030f2a00000)
libz.so.1 => /lib64/libz.so.1 (0x00000030e8800000)
libc.so.6 => /lib64/libc.so.6 (0x00000030e7800000)
/lib64/ld-linux-x86-64.so.2 (0x00000030e7400000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000030e8000000)
查看结果显示 ： libpcre.so.1 => not found ，同时注意lib库的路径，有/lib/* 和 /lib64/* 之分。比如上面的是 /lib64/*,这个和下面解决问题时创建的软连接有关系

解决办法：
1、首先确认已经安装好pcre 软件（nginx 依赖该软件）
2、创建软连接

（1）对于/lib/* 32位系统来说：
#查看lib库（ps: 也有可能 pcre lib文件在目录：/usr/local/lib/）
#  ls /lib/ |grep pcre
libpcre.so.0
libpcre.so.0.0.1
#添加软连接
# ln -s /lib/libpcre.so.0.0.1 /lib/libpcre.so.1

（2）对于/lib64/* 64位系统来说：
#查看lib库（ps: 也有可能 pcre lib文件在目录：/usr/local/lib64/）
#  ls /lib64/ |grep pcre
libpcre.so.0
libpcre.so.0.0.1
#添加软连接
# ln -s /lib64/libpcre.so.0.0.1 /lib64/libpcre.so.1

检查是否启动

[root@www local]# netstat -ano|grep 80
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      off (0.00/0/0)

7.设置开机自动启动

首先，在linux系统的/etc/init.d/目录下创建nginx文件，使用如下命令：vim /etc/init.d/nginx

在脚本中添加

#!/bin/bash  
# nginx Startup script for the Nginx HTTP Server  
# it is v.0.0.2 version.  
# chkconfig: - 85 15  
# description: Nginx is a high-performance web and proxy server.  
#              It has a lot of features, but it's not for everyone.  
# processname: nginx  
# pidfile: /var/run/nginx.pid  
# config: /usr/local/nginx/conf/nginx.conf  
nginxd=/usr/local/nginx/sbin/nginx  
nginx_config=/usr/local/nginx/conf/nginx.conf  
nginx_pid=/var/run/nginx.pid  
RETVAL=0  
prog="nginx"  
# Source function library.  
. /etc/rc.d/init.d/functions  
# Source networking configuration.  
. /etc/sysconfig/network  
# Check that networking is up.  
[ ${NETWORKING} = "no" ] && exit 0  
[ -x $nginxd ] || exit 0  
# Start nginx daemons functions.  
start() {  
if [ -e $nginx_pid ];then  
   echo "nginx already running...."  
   exit 1  
fi  
   echo -n $"Starting $prog: "  
   daemon $nginxd -c ${nginx_config}  
   RETVAL=$?  
   echo  
   [ $RETVAL = 0 ] && touch /var/lock/subsys/nginx  
   return $RETVAL  
}  
# Stop nginx daemons functions.  
stop() {  
        echo -n $"Stopping $prog: "  
        killproc $nginxd  
        RETVAL=$?  
        echo  
        [ $RETVAL = 0 ] && rm -f /var/lock/subsys/nginx $nginx_pid   
}  
# reload nginx service functions.  
reload() {  
    echo -n $"Reloading $prog: "  
    #kill -HUP `cat ${nginx_pid}`  
    killproc $nginxd -HUP  
    RETVAL=$?  
    echo  
}  
# See how we were called.  
case "$1" in  
start)  
        start  
        ;;  
stop)  
        stop  
        ;;  
reload)  
        reload  
        ;;  
restart)  
        stop  
        start  
        ;;  
status)  
        status $prog  
        RETVAL=$?  
        ;;  
*)  
        echo $"Usage: $prog {start|stop|restart|reload|status|help}"  
        exit 1  
esac  
exit $RETVAL

请根据自己服务器的实际情况进行修改。

nginxd=/usr/local/nginx/sbin/nginx

nginx_config=/usr/local/nginx/conf/nginx.conf
修改文件的权限

chmod a+x /etc/init.d/nginx
将nginx加入到rc.local文件中，这样开机的时候nginx就默认启动了

vi /etc/rc.local
添加
/etc/init.d/nginx start
保存并退出
现在就可以用service nginx stop/start/status
看能否关闭/启动nginx
测试机器的话，可以重启下服务器，验证下是否实现了nginx的自启动。

8.nginx配置文件

修改两台机器的配置文件 vim /usr/local/nginx/conf/nginx.conf

修改前的配置：

nginx1.2.8+keepalived1.2.24实现高可用负载均衡_第3张图片

原配置文件.png

修改后的配置：

nginx1.2.8+keepalived1.2.24实现高可用负载均衡_第4张图片

修改后的配置.png

http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name  to: $upstream_addr $upstream_status $request upstream_response_time $upstream_response_time msec $msec request_time $request_time';

    access_log  logs/access.log  upstreamlog;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;
    #

    # mhn
    upstream webservices {
        server 10.0.1.227:9200  weight=1 max_fails=3 fail_timeout=10s;
        server 10.0.1.45:9200 weight=1 max_fails=3 fail_timeout=10s;
    }

    server {
        listen       80;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            #root   html;
            #index  index.html index.htm;

            # mhn
            proxy_pass http://webservices;
            proxy_set_header X-Real-IP $remote_addr;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

重新加载配置文件(可以不用重启nginx)：/usr/local/nginx/sbin/nginx -s reload

查看nginx日志文件：vim /usr/local/nginx/logs/access.log，这里其实是更改了nginx的默认日志格式，目的是看到nginx转发后的访问ip地址的变化，可以发现10.0.1.227和10.0.1.45机器来回切换，做到了负载均衡。
其实要是观察时间的话，有同一时间访问两个地址的情况（直接在页面点击刷新可以重现），额……问到我了，我也是有些懵逼的，不过手动输入ip地址，按下回车，就只会访问到一个地址，这里只是描述了下操作步骤，具体原因，后续查查资料

[root@www logs]# ll /usr/local/nginx/logs/
total 164
-rw-r--r-- 1 root root 153172 Nov 10 20:04 access.log
-rw-r--r-- 1 root root   4790 Nov 10 14:14 error.log
-rw-r--r-- 1 root root      6 Nov 11 12:50 nginx.pid
[root@www logs]# vim access.log 


多次访问当前机器的ip地址，默认为80端口，可以发现出现的日志格式为：
[10/Nov/2017:17:57:44 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.227:9200 200 GET / HTTP/1.1 upstream_response_time 0.210 msec 1510307864.842 request_time 0.210
[10/Nov/2017:17:57:44 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.45:9200 200 GET /favicon.ico HTTP/1.1 upstream_response_time 0.004 msec 1510307864.981 request_time 0.004
[10/Nov/2017:17:57:46 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.227:9200 200 GET / HTTP/1.1 upstream_response_time 0.006 msec 1510307866.625 request_time 0.006
[10/Nov/2017:17:57:46 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.45:9200 200 GET /favicon.ico HTTP/1.1 upstream_response_time 0.004 msec 1510307866.710 request_time 0.004
[10/Nov/2017:17:58:27 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.227:9200 200 GET / HTTP/1.1 upstream_response_time 0.008 msec 1510307907.580 request_time 0.008
[10/Nov/2017:17:58:27 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.45:9200 200 GET /favicon.ico HTTP/1.1 upstream_response_time 0.005 msec 1510307907.746 request_time 0.005
[10/Nov/2017:17:58:49 +0800] 10.0.0.222 - - - localhost  to: - - - upstream_response_time - msec 1510307929.048 request_time 0.000
[10/Nov/2017:18:54:19 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.227:9200 200 GET / HTTP/1.1 upstream_response_time 0.007 msec 1510311259.224 request_time 0.007
[10/Nov/2017:18:54:19 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.45:9200 200 GET /favicon.ico HTTP/1.1 upstream_response_time 0.009 msec 1510311259.398 request_time 0.009

[10/Nov/2017:18:55:03 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.227:9200 200 GET / HTTP/1.1 upstream_response_time 0.005 msec 1510311303.381 request_time 0.005
[10/Nov/2017:18:55:12 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.45:9200 200 GET / HTTP/1.1 upstream_response_time 0.004 msec 1510311312.303 request_time 0.004
[10/Nov/2017:18:55:15 +0800] 10.0.0.222 - - - localhost  to: 10.0.1.227:9200 200 GET / HTTP/1.1 upstream_response_time 0.005 msec 1510311315.297 request_time 0.005

Keepalived安装

1.安装

[root@www src]# cd /usr/local/src
[root@www src]# wget http://www.keepalived.org/software/keepalived-1.2.24.tar.gz
[root@www src]# tar -zxvf keepalived-1.2.24.tar.gz
[root@www src]# cd keepalived-1.2.24

这一步会出现问题，见（1）（2）
[root@www src]# ./configure --prefix=/usr/local/keepalived

这一步也会出现问题，见（3）
[root@www src]# make && make install

安装遇到的问题解决方式：
（1）系统出现警告信息“*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.”，具体日志如下：

Keepalived configuration
------------------------
Keepalived version       : 1.3.5
Compiler                 : gcc
Preprocessor flags       : 
Compiler flags           : -Wall -Wunused -Wstrict-prototypes -Wextra -g -O2
Linker flags             : 
Extra Lib                :  -lcrypto  -lssl 
Use IPVS Framework       : Yes
IPVS use libnl           : No
IPVS syncd attributes    : No
IPVS 64 bit stats        : No
fwmark socket support    : Yes
Use VRRP Framework       : Yes
Use VRRP VMAC            : Yes
Use VRRP authentication  : Yes
With ip rules/routes     : Yes
SNMP vrrp support        : No
SNMP checker support     : No
SNMP RFCv2 support       : No
SNMP RFCv3 support       : No
DBUS support             : No
SHA1 support             : No
Use Debug flags          : No
Stacktrace support       : No
Memory alloc check       : No
libnl version            : None
Use IPv4 devconf         : No
Use libiptc              : No
Use libipset             : No
init type                : upstart
Build genhash            : Yes
Build documentation      : No
 
*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

解决方案一（在线安装）：yum -y install libnl libnl-devel

解决方案二（离线安装）：离线安装libnl-devel包，从网上下载libnl-devel-1.1.4-2.el6.x86_64.rpm包，执行rpm -ivh libnl-devel-1.1.4-2.el6.x86_64.rpm命令。

[root@www src]# wget http://mirror.centos.org/centos/6/os/x86_64/Packages/libnl-devel-1.1.4-2.el6.x86_64.rpm
####执行wget命令是从另外一台可以连接外网的服务器上下载libnl-devel-1.1.4-2.el6.x86_64.rpm包的命令，可以忽略，只是为大家提供个下载libnl-devel地址####
 
[root@www src]# rpm -ivh libnl-devel-1.1.4-2.el6.x86_64.rpm 
Preparing...                ########################################### [100%]
    package libnl-devel-1.1.4-2.el6.x86_64 is already installed

安装完成以后，重新执行configure命令

（2）系统出现错误信息“configure: error: libnfnetlink headers missing”，具体日志如下：

checking for nl_socket_modify_cb in -lnl... yes
checking for linux/rtnetlink.h... yes
checking libnfnetlink/libnfnetlink.h usability... no
checking libnfnetlink/libnfnetlink.h presence... no
checking for libnfnetlink/libnfnetlink.h... no
configure: error: libnfnetlink headers missing

解决方案一（在线安装）：yum install -y libnfnetlink-devel

解决方案二（离线安装）：离线安装libnl-devel包，从网上下载libnl-devel-1.1.4-2.el6.x86_64.rpm包，执行rpm -ivh libnl-devel-1.1.4-2.el6.x86_64.rpm命令。

#Linux执行wget下载rpm包（离线安装libnfnetlink包时所需rpm包，执行wget下载libnfnetlink rpm包命令，此步可以忽略，只是为大家提供个下载libnfnetlink地址），具体命令如下：
wget ftp://mirror.switch.ch/mirror/centos/6/os/x86_64/Packages/libnfnetlink-1.0.0-1.el6.x86_64.rpm
wget ftp://mirror.switch.ch/mirror/centos/6/os/x86_64/Packages/libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm

#执行rpm -ivh libnfnetlink-1.0.0-1.el6.x86_64.rpm命令安装libnfnetlink-1.0.0-1.el6.x86_64.rpm包，具体如下：
[root@www src]# rpm -ivh libnfnetlink-1.0.0-1.el6.x86_64.rpm 
Preparing...                ########################################### [100%]
   1:libnfnetlink           ########################################### [100%]

#执行rpm -ivh libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm命令安装libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm包，具体如下：
[root@www src]# rpm -ivh libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:libnfnetlink-devel     ########################################### [100%]

安装完成以后，重新执行configure命令
（3）编译keepalived不会出现上述警告和其他错误，执行make && make install命令进行安装，若出现一下错误：

Making all in lib
make[1]: Entering directory `/root/keepalived-1.2.24/lib'
make  all-am
make[2]: Entering directory `/root/keepalived-1.2.24/lib'
make[2]: Leaving directory `/root/keepalived-1.2.24/lib'
make[1]: Leaving directory `/root/keepalived-1.2.24/lib'
Making all in keepalived
make[1]: Entering directory `/root/keepalived-1.2.24/keepalived'
Making all in core
make[2]: Entering directory `/root/keepalived-1.2.24/keepalived/core'
  CC       namespaces.o
namespaces.c: In function ‘setns’:
namespaces.c:184: error: ‘SYS_setns’ undeclared (first use in this function)
namespaces.c:184: error: (Each undeclared identifier is reported only once
namespaces.c:184: error: for each function it appears in.)
make[2]: *** [namespaces.o] Error 1
make[2]: Leaving directory `/root/keepalived-1.2.24/keepalived/core'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/keepalived-1.2.24/keepalived'
make: *** [all-recursive] Error 1

升级了下glib解决了，大家可以试试，yum update glib*

2.配置

查看配置文件目录的结构：

安装完成后，进入安装目录的etc目录下，将keepalived相应的配置文件拷贝到系统相应的目录当中。keepalived启动时会从/etc/keepalived目录下查找keepalived.conf配置文件，如果没有找到则使用默认的配置。/etc/keepalived目录安装时默认是没有安装的，需要手动创建。

nginx1.2.8+keepalived1.2.24实现高可用负载均衡_第5张图片

image.png

[root@www local]# mkdir /etc/keepalived
[root@www local]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
[root@www local]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/keepalived
[root@www local]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
[root@www local]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

这样就可以执行service keepalived [start | stop | reload | restart ]命令
keepalived正常运行后，会启动3个进程，其中一个是父进程，负责监控其子进程。一个是vrrp子进程，另外一个是checkers子进程

ps -ef | grep keepalived
root       831     1  0 11:22 ?        00:00:00 keepalived -D
root       840   831  0 11:22 ?        00:00:00 keepalived -D
root       841   831  0 11:22 ?        00:00:00 keepalived -D

（1）若是启动报“Starting keepalived: /bin/bash: keepalived: command not found”错误，具体日志如下：

[root@yoodb /]# service keepalived start
正在启动 keepalived：/bin/bash: keepalived: command not found
                                                           [失败]

执行cp /usr/local/keepalived/sbin/keepalived /usr/sbin/命令就好了，也就是上面的第四条命令

（2）使用service keepalived start命令启动服务时，默认会将/etc/sysconfig/keepalived文件中KEEPALIVED_OPTIONS参数作为keepalived服务启动时的参数，并从/etc/keepalived/目录下加载keepalived.conf配置文件，或用-f参数指定配置文件的位置
我们可以发现默认的“-D”参数，代表着keepalived的启动日志会在/var/log/message文件中
运行keepalived --help可以查看启动时的可选参数，这些可选参数都可以配置

# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp               -P    Only run with VRRP subsystem.
# --check              -C    Only run with Health-checker subsystem.
# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.
# --dump-conf          -d    Dump the configuration data.
# --log-detail         -D    Detailed log messages.日志默认输出在/var/log/message文件中
# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)
#
KEEPALIVED_OPTIONS="-D"

3.keepalived.conf配置文件 `vi /etc/keepalived/keepalived.conf`

nginx1.2.8+keepalived1.2.24实现高可用负载均衡_第6张图片

keepalived配置文件.png

! Configuration File for keepalived

global_defs {
   notification_email {
    [email protected]
   }
   notification_email_from [email protected]
   smtp_server smtp.gmail.com
   smtp_connect_timeout 30
}
vrrp_script check_nginx {                   ######定义监控nginx的脚本
    script "/usr/local/script/check_nginx.sh"
    interval 2                            ######监控时间间隔
    weight 2                              ######负载参数
}
vrrp_instance vrrptest {                 ######定义vrrptest实例
        state MASTER              ######服务器状态
        interface eth0                     ######使用的接口
        virtual_router_id 51               ######虚拟路由的标志，一组lvs的虚拟路由标识必须相同，这样才能切换
        priority 150                       ######服务启动优先级，值越大，优先级越高，BACKUP 不能大于MASTER
        advert_int 1                        ######服务器之间的存活检查时间

        authentication {
                auth_type PASS                     ######认证类型
                auth_pass ufsoft       ######认证密码，一组lvs 服务器的认证密码必须一致
        }
        track_script {                              ######执行监控nginx进程的脚本
                check_nginx
        }
        virtual_ipaddress {                         ######虚拟IP地址
                10.0.0.79

        }
}

注意：

需要修改下“从”的priority的值，改成比“主”的小。
“主”的state为MASTER，“从”的state为BACKUP
两台机器的virtual_router_id必须一致

4.查看下check_nginx.sh脚本`vim /usr/local/script/check_nginx.sh`

if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
 then
    echo "keepalived stop!!!"  
    service keepalived stop
 else
    echo "nginx is running"  
fi

说明：grep -v grep 指在搜索结果中过滤掉包含有grep字符的行
记得更改权限：
chmod +x /usr/local/script/check_nginx.sh

或者加入到作业计划（每一分钟执行一次），其实是不需要的，因为keepalived已经会执行脚本了
crontab -e
*/1 * * * * /usr/local/script/check_nginx.sh >>/usr/local/script/check_nginx.log
!wq

5.启动keepalived进行验证 `service keepalived start`

使用命令ip addr来查看vip

[root@www local]# ip addr
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:52:a9:87 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.78/16 brd 10.0.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.79/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe52:a987/64 scope link 
       valid_lft forever preferred_lft forever
[root@www local]#

关掉主的nginx,看到vip漂移到了从，且仍然能通过虚拟IP均衡地访问web服务器。

THE END

参考文档

安装部署nginx
Nginx启动错误:error while loading shared libraries: libpcre.so.1
nginx+keepalive实现高可用负载均衡
Keepalived安装与配置
Nginx keepalived实现高可用负载均衡详细配置步骤