实现Haproxy+keepalived高可用集群转发
目录
实现Haproxy+keepalived高可用集群转发
一 网络图及服务器规划
二 安装服务
1 安装keepalived
2 安装haproxy
3 安装nginx
三 服务配置
1. keepalived双主配置
2. haproxy配置
3. nginx配置
四 测试验证
实现Haproxy+keepalived高可用集群转发
如有错误，请留言指正。
一 网络图及服务器规划
| 服务器名称 | 操作系统 | IP地址 | VIP(MASTER) | 应用名称 |
| zlc1.com | Ubuntu 18.0.4 | 172.16.0.101 | 172.16.0.10 172.16.0.11 |
keepalived haproxy |
| zlc2.com | Ubuntu 18.0.4 | 172.16.0.102 | 172.16.0.12 172.16.0.13 |
keepalived haproxy |
| zlc3.com | Centos 7 | 172.16.0.201 | nginx | |
| zlc4.com | Centos 7 | 172.16.0.202 | nginx |
二 安装服务
1 安装keepalived
1.1 yum安装
• keepalived环境准备:
• 各节点时间必须同步
ntp, chrony
• 关闭selinux和防火墙
• 各节点之间可通过主机名互相通信（对KA并非必须）
建议使用/etc/hosts文件实现，如果DNS出现问题会使稳定性大打折扣
• 各节点之间的root用户可以基于密钥认证的ssh服务完成互相通信（对KA并非必须）
• Keepalived安装：
# yum install keepalived (CentOS)
# apt-get install keepalived (Ubuntu)
1.2 编译安装
# wget http://www.keeplived.org/software/keepalived-2.0.19.tar.gz
# yum install libnfnetlink-devel libnfnetlink ipvsadm libnl libnl-devel libnl3 libnl3-devel \
lm_sensors-libs net-snmp-agent-libs net-snmp-libs openssh-server openssh-clients openssl \
openssl-devel automake iproute# ./configure --prefix=/usr/local/keepalived --disable-fwmark
#make && make install
#cp keepalived-2.0.19/keepalived/keepalived.service /usr/lib/systemd/system/keepalived.service
2 安装haproxy
编译安装HAProxy 2.0 LTS版本，更多源码包下载地址：http://www.haproxy.org/download/
（1）由于centos自带的lua版本比较低并不符合HAProxy要求的lua最低版本(5.3)的要求，因此需要编译安装较新版本的lua环境，然后才能编译安装HAProxy，过程如下：
# yum install libtermcap-devel ncurses-devel libevent-devel readline-devel
# wget http://www.lua.org/ftp/lua-5.3.5.tar.gz
# tar xvf lua-5.3.5.tar.gz
# cd lua-5.3.5
# make linux test
# pwd
/usr/local/src/lua-5.3.5
# lua -v #当前系统版本
Lua 5.1.4 Copyright (C) 1994-2008 Lua.org, PUC-Rio
# ./src/lua -v #编译安装的版本
Lua 5.3.5 Copyright (C) 1994-2018 Lua.org, PUC-Rio
（2）编译安装HAProxy：
# pwd
/usr/local/src
# tar xvf haproxy-2.0.4.tar.gz
# yum install gcc gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel
systemd-devel net-tools vim iotop bc zip unzip zlib-devel lrzsz tree screen lsof
tcpdump wget ntpdate #安装编译环境
注：Ubuntu需额外安装包libsystemd-dev
#HAProxy 1.8及1.9版本编译参数：
make ARCH=x86_64 TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1
USE_CPU_AFFINITY=1 PREFIX=/usr/local/haproxy
#HAProxy 2.0编译参数：
# make ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
USE_SYSTEMD=1 USE_CPU_AFFINITY=1 USE_LUA=1 LUA_INC=/usr/local/src/lua-5.3.5/src/
LUA_LIB=/usr/local/src/lua-5.3.5/src/ PREFIX=/usr/local/haproxy
# make install PREFIX=/usr/local/haproxy
# cp haproxy /usr/sbin/
（3）HAProxy启动脚本：
# cat /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target
[Service]
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/lib/haproxy/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID
[Install]
WantedBy=multi-user.target
3 安装nginx
(1) 准备编译安装的基础环境：
yum install -y vim lrzsz tree screen psmisc lsof tcpdump wget ntpdate gcc gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel systemd-devel nettools iotop bc zip unzip zlib-devel bash-completion nfs-utils automake libxml2 libxml2-devel libxslt libxslt-devel perl perl-ExtUtils-Embed
(2) 安装nginx
官方源码包下载地址：
https://nginx.org/en/download.html[root@s2 ~]# cd /usr/local/src/
[root@s2 src]# wget https://nginx.org/download/nginx-1.16.1.tar.gz
[root@s2 src]# tar xf nginx-1.16.1.tar.gz
[root@s2 src]# cd nginx-1.16.1/[root@s2 nginx-1.16.1]#./configure --prefix=/apps/nginx \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_realip_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-pcre \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module[root@s2 nginx-1.16.1]# make #编译步骤，根据Makefile文件生成相应的模块
[root@s2 nginx-1.16.1]# make install #创建目录，并将生成的模块和文件复制到相应的目录：
[root@s2 nginx-1.16.1]# useradd nginx -s /sbin/nologin -u 2000 #以普通用户启动nginx
[root@s2 nginx-1.16.1]# chown nginx.nginx -R /apps/nginx/（3）创建Nginx自启动脚本：
[root@s1 ~]# cat /usr/lib/systemd/system/nginx.service
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target
[Service]
Type=forking
PIDFile=/run/nginx.pid
# Nginx will fail to start if /run/nginx.pid already exists but has the wrong
# SELinux context. This might happen when running `nginx -t` from the cmdline.
# https://bugzilla.redhat.com/show_bug.cgi?id=1268621
ExecStartPre=/usr/bin/rm -f /run/nginx.pid
ExecStartPre=/apps/nginx/sbin/nginx -t
ExecStart=/apps/nginx/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=process
PrivateTmp=true
[Install]
WantedBy=multi-user.target
三 服务配置
1. keepalived双主配置
1.1 zlc1配置
# cat /etc/keepalived/keepalived.conf
global_defs {
notification_email {
#keepalived 发生故障切换时邮件发送的对象，可以按行区分写多个
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ha1.example.com
# 所有报文都检查比较消耗性能，此配置为如果收到的报文和上一个报文是同一个路由器则跳过检查报文中的源地址
vrrp_skip_check_adv_addr
#严格遵守VRRP协议,不允许状况:1,没有VIP地址,2.配置了单播邻居,3.在VRRP版本2中有IPv6地址.
# vrrp_strict
vrrp_garp_interval 0 #ARP报文发送延迟
vrrp_gna_interval 0 #消息发送延迟
# 默认组播IP地址， 224.0.0.0到239.255.255.255
# vrrp_mcast_group4 224.0.0.18
#yum安装的keepalived配置的虚拟ip不能被访问，自动配置了防火墙策略，加入该参数，则不再添加该策略
vrrp_iptables
}
# VIP转移策略配置，通过判断某脚本的返回值，执行权重的增减，master或backup其中一方配置即可，
# 经确认，master检测脚本返回值非0时执行权重减；backup检测脚本返回值为0时执行权重增。
vrrp_script chk_haproxy {
script "/etc/keepalived/chk_haproxy.sh"
interval 1
weight -80
fall 3
rise 5
timeout 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 80
priority 100
advert_int 1
# 单播通知
unicast_src_ip 172.16.0.101
unicast_peer {
172.16.0.102
}
authentication {
auth_type PASS
auth_pass 1111qwer
}
#调用检查
track_script {
chk_haproxy
}
virtual_ipaddress {
172.16.0.10/16 dev eth0 label eth0:0
172.16.0.11/16 dev eth0 label eth0:1
}
}
vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 88
priority 80
advert_int 1
unicast_src_ip 172.16.0.101
unicast_peer {
172.16.0.102
}
authentication {
auth_type PASS
auth_pass 1111qwer
}
virtual_ipaddress {
172.16.0.12/16 dev eth0 label eth0:0
172.16.0.13/16 dev eth0 label eth0:1
}
}
[root@s1 ~]# cat /etc/keepalived/chk_haproxy.sh
#!/bin/bash
/usr/bin/killall -0 haproxy
[root@s1 ~]# chmod a+x /etc/keepalived/chk_haproxy.sh
1.2 zlc2配置
# cat /etc/keepalived/keepalived.conf
global_defs {
notification_email {
#keepalived 发生故障切换时邮件发送的对象，可以按行区分写多个
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id ha1.example.com
# 所有报文都检查比较消耗性能，此配置为如果收到的报文和上一个报文是同一个路由器则跳过检查报文中的源地址
vrrp_skip_check_adv_addr
#严格遵守VRRP协议,不允许状况:1,没有VIP地址,2.配置了单播邻居,3.在VRRP版本2中有IPv6地址.
# vrrp_strict
vrrp_garp_interval 0 #ARP报文发送延迟
vrrp_gna_interval 0 #消息发送延迟
# 默认组播IP地址， 224.0.0.0到239.255.255.255
# vrrp_mcast_group4 224.0.0.18
#yum安装的keepalived配置的虚拟ip不能被访问，自动配置了防火墙策略，加入该参数，则不再添加该策略
vrrp_iptables
}
# VIP转移策略配置，通过判断某脚本的返回值，执行权重的增减，master或backup其中一方配置即可，
# 经确认，master检测脚本返回值非0时执行权重减；backup检测脚本返回值为0时执行权重增。
vrrp_script chk_haproxy {
script "/etc/keepalived/chk_haproxy.sh"
interval 1
weight -80
fall 3
rise 5
timeout 2
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 80
priority 80
advert_int 1
# 单播通知
unicast_src_ip 172.16.0.102
unicast_peer {
172.16.0.101
}
authentication {
auth_type PASS
auth_pass 1111qwer
}
virtual_ipaddress {
172.16.0.10/16 dev eth0 label eth0:0
172.16.0.11/16 dev eth0 label eth0:1
}
}
vrrp_instance VI_2 {
state MASTER
interface eth0
virtual_router_id 88
priority 100
advert_int 1
unicast_src_ip 172.16.0.102
unicast_peer {
172.16.0.101
}
authentication {
auth_type PASS
auth_pass 1111qwer
}
#调用检查
track_script {
chk_haproxy
}
virtual_ipaddress {
172.16.0.12/16 dev eth0 label eth0:0
172.16.0.13/16 dev eth0 label eth0:1
}
}
2. haproxy配置
(1) 配置文件
# mkdir /etc/haproxy
# cat /etc/haproxy/haproxy.cfg
global
maxconn 100000
chroot /usr/local/haproxy
stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin
uid 99
gid 99
daemon
# 多进程模式，下边五行若取消注释，上方的socket也要修改成对应的进程数
#nbproc 4
#cpu-map 1 0
#cpu-map 2 1
#cpu-map 3 2
#cpu-map 4 3
pidfile /var/lib/haproxy/haproxy.pid
log 127.0.0.1 local3 info
defaults
option http-keep-alive
option forwardfor
maxconn 100000
mode http
timeout connect 300000ms
timeout client 300000ms
timeout server 300000ms
listen stats
mode http
bind 0.0.0.0:9999
stats enable
log global
stats uri /haproxy-status
status auth haadmin:12345
listen web_port
bind 172.16.0.10:80 ,172.16.0.11:80, 172.16.0.12:80 ,172.16.0.13:80
# 注：配置内核参数：net.ipv4.ip_nonlocal_bind=1 使haproxy可监听在本机没有的ip地址上
mode http
log global
server web1 172.16.0.201:8080 check inter 3000 fall 2 rise 5
server web2 172.16.0.202:8080 check inter 3000 fall 2 rise 5
(2) 启动haproxy
# mkdir /var/lib/haproxy
# chown 99.99 /var/lib/haproxy/ -R
# systemctl start haproxy
# systemctl enable haproxy
# systemctl status haproxy
3. nginx配置
[root@s2 ~]# grep -v "#" /apps/nginx/conf/nginx.conf | grep -v "^$"
#全局配置段
user nginx nginx;
worker_processes 1;
events {
worker_connections 1024;
}http {
include mime.types;
default_type application/octet-stream;
sendfile on；
keepalive_timeout 65;
server {
listen 8080;
server_name zlc3.com;
location / {
root html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
添加测试主页
echo $(hostname) > /apps/nginx/html/index.html
四 测试验证
1.停止keepalived服务
（1）循环curl http://172.16.0.10，停止zlc1的keepalived服务，查看zlc1和zlc2的ip变化，查看curl是否出现几秒的无响应状态
( 2 ) 循环curl http://172.16.0.12 ，停止zlc2的keepalived服务，查看zlc1和zlc2的ip变化，查看curl是否出现几秒的无响应状态
2. 停止haproxy服务
（1）循环curl http://172.16.0.11，停止zlc1的haproxy服务，查看zlc1和zlc2的ip变化，查看zlc1后台日志权重变化，查看curl是否出现几秒的无响应状态
（2）循环curl http://172.16.0.13，停止zlc2的haproxy服务，查看zlc1和zlc2的ip变化，查看zlc2后台日志权重变化，查看curl是否出现几秒的无响应状态