本文描述的是haproxy做代理，负载均衡，haproxy状态查看，后端主机的上下线，keepalived高可用
一、前提条件
1.1 至少两台服务器 操作系统为centos7
IP：192.168.238.138/24机器名：ceph4
IP：192.168.238.139/24机器名：ceph5
1.2 部署软件（两台都有）：
Keepalived高可用
Haproxy1.7.9反向代理
Apache HTTP后端主机
1.3 在两台主机设置分别机器名
#hostnamectl ceph4
#hostnamectl ceph4
1.4编辑vi /etc/hosts中加入(两台都有)
#加入本机的域名解析
192.168.238.138 ceph4
192.168.238.139 ceph5
1.5 关闭防火墙，selinux(两台都有)
#systemctl stop firewalld#停止正在运行的防火墙
#systemctl disabled firewalld#禁止开机启动
#setenforce 0#临时禁止selinux
#sed -i "s/^SELINUX\=enforcing/SELINUX\=disabled/g"/etc/selinux/config#永久禁止
二、安装配置后端软件apache http
2.1 安装http软件（两台都有）
#yum install httpd -y
2.2 编辑配置文件改监听的端口
安装完编辑vi /etc/httpd/conf/httpd.conf
Listen 8080 #监听端口改为8080，可以不修改，我的是主机的80端口已经被占用
2.3 配置访问的页面
Ceph4：
#echo 'ceph4' >/var/www/html/index.html
Ceph5：
#echo 'ceph5' >/var/www/html/index.html
2.4 启动http并测试
#systemctl start httpd
#curl ceph4:8080
ceph4#结果不同，以方便之后的测试
#curl ceph5:8080
ceph5
三、Haproxy安装部署
下边的操作两个节点都需要安装
3.1 相关系统包的安装
#yum install -y gcc glibc gcc-c++ make screen tree lrzsz
3.2 Haproxy安装
#mkdir /soft#创建目录
#cd /soft/
#wgethttp://www.haproxy.org/download/1.7/src/haproxy-1.7.9.tar.gz#下载安装包
#tar xf haproxy-1.7.9.tar.gz#解压
#cd haproxy-1.7.9
#make TARGET=linux2628 PREFIX=/usr/local/haproxy1.7.9#编译
#make install#安装
install -d "/usr/local/sbin"
install haproxy"/usr/local/sbin"
install -d "/usr/local/share/man"/man1
install -m 644 doc/haproxy.1 "/usr/local/share/man"/man1
install -d "/usr/local/doc/haproxy"
for x in configuration management architecture cookie-options luaWURFL-device-detection proxy-protocol linux-syn-cookies network-namespacesDeviceAtlas-device-detection 51Degrees-device-detectionnetscaler-client-ip-insertion-protocol close-options SPOE intro; do \
install -m 644doc/$x.txt "/usr/local/doc/haproxy" ; \
#cp /usr/local/sbin/haproxy /usr/sbin/#启动文件
#haproxy-v#查看安装结果
HA-Proxy version 1.7.9 2017/08/18
Copyright 2000-2017 Willy [email protected]
创建haproxy启动脚本
#cp examples/haproxy.init /etc/init.d/haproxy
#/etc/init.d/haproxy start#启动
创建需要的相关的目录
#useradd -r haproxy
#mkdir /etc/haproxy
#mkdir /var/lib/haproxy
#mkdir /var/run/haproxy
编辑haproxy配置文件
#vi /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local3 info
chroot /var/lib/haproxy
maxconn10000#设置允许的最大连接数，需要考虑ulimit -n的限制
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
frontend front#前端
mode http
bind *:8088#这里的端口为8088，也可以是其他为占用的端口
stats uri /haproxy?stats
default_backend default_backend
backend default_backend#后端
#source cookie SERVERID
option forwardfor header X-REAL-IP
option httpchk GET /index.html#检查的url
balance roundrobin
server ceph5 192.168.238.139:8080 check inter 2000 rise 3 fall 3 weight1
server ceph4 192.168.238.138:8080 check inter 2000 rise 3 fall 3 weight1
日志设置
#sed -i 's@\#\$ModLoad imudp@\$ModLoad imudp@g' /etc/rsyslog.conf
#sed-i 's@\#\$UDPServerRun514@\$UDPServerRun 514@g' /etc/rsyslog.conf
#echo "local3.*/var/log/haproxy.log" >> /etc/rsyslog.conf
启动：
#/etc/init.d/haproxy start
Startinghaproxy (via systemctl):[OK]
测试：
# ceph5的haproxy配置正常
[root@ceph4 ~]# curlceph5:8088
ceph5
[root@ceph4 ~]# curlceph5:8088
ceph4
# ceph4的haproxy配置正常
[root@ceph4 ~]# curlceph4:8088
ceph5
[root@ceph4 ~]# curlceph4:8088
ceph4
看到访问url的结果是两台服务器轮换相应。
状态管理页面
在浏览器访问http://192.168.238.138:8088/haproxy?stats，查看状态
3.3 Haproxy动态维护（两点都需要）
在配置文件的global下添加socket文件
stats socket /var/lib/haproxy/haproxy.sockmode 600 level admin
stats timeout 2m
安装socat
#yum install -y socat
查看haproxy的帮助
#echo "help" |socat stdio /var/lib/haproxy/haproxy.sock
查看info状态信息，可以通过zabbix来监控相关状态值
#echo "show info" |socat stdio/var/lib/haproxy/haproxy.sock
Name: HAProxy
Version: 1.7.9
Release_date: 2017/08/18
Nbproc: 1
Process_num: 1
Pid: 5145
Uptime: 0d 0h03m34s
Uptime_sec: 214
Memmax_MB: 0
PoolAlloc_MB: 0
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 20033
Maxsock: 20033
Maxconn: 10000
Hard_maxconn: 10000
CurrConns: 0
CumConns: 4
CumReq: 4
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 0
SessRate: 0
SessRateLimit: 0
MaxSessRate: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
Tasks: 7
Run_queue: 1
Idle_pct: 100
node: ceph4
haproxy维护模式（主机上下线）
在ceph4上做测试，下线default_backend下的ceph4主机
#echo "disable server default_backend/ceph4 " |socat stdio/var/lib/haproxy/haproxy.sock
注：ceph4已经不在线
上线default_backend下的ceph4
#echo "enable server default_backend/ceph4 " |socat stdio/var/lib/haproxy/haproxy.sock
注：ceph4恢复
3.4 Haproxy生产使用建议
haproxy的本地端口会出现用尽情况，解决方案如下4条
1.更改local的端口范围,调整内核参数
#cat /proc/sys/net/ipv4/ip_local_port_range
3276861000
2.调整timewait的端口复用，设置为1
#cat /proc/sys/net/ipv4/tcp_tw_reuse
1
3.调整tcp_wait的时间，不建议修改
#cat /proc/sys/net/ipv4/tcp_fin_timeout
60
4.最佳方案：增加多个ip，端口数量就足够
四、Keepalived
Mail配置使用
4.1安装mailx邮件服务
yum install mailx -y
#配置文件追加信息（/etc/mail.rc）
vim /etc/mail.rc
#发件人信息
set [email protected]#发件人邮箱地址(163设置得开起允许代理)
set smtp=smtp.163.com#smtp地址
[email protected]#邮箱用户名，不用加域名
set smtp-auth-password=******#邮箱密码（邮件密码是smtp代理授权码）
set smtp-auth=login#邮箱验证方式
#测试发送
echo "hello world" | mail -s"hello"[email protected]@qq.com#会看到测试邮件信息，可以发送多个邮件
#echo "邮件内容"
| mail -s "标题"邮箱地址
#最好把你的发送邮件地址加入你接收邮箱的白名单，不然发多了可能被认为发送垃圾邮件而被163拒绝，这是真的
4.2.安装配置keepalived（两节点都需要）
# yum install -y keepalived
# keepalived -v#查看版本
Keepalived v1.3.5 (03/19,2017), git commitv1.3.5-6-g6fa32f2
在/etc/keepalived下建立文件如下（两节点）：
# ls
check_haproxy.shcheck_haproxy_url.shdown.shkeepalived.confvrrp.sh
#主要是一些脚本和keepalived配置文件
# vi check_haproxy.sh#检测haproxy进程是否村子，不存在的话重启
#!/bin/bash
counter=$(ps -C haproxy --no-heading|wc -l)
if [ "${counter}" = "0"]; then
/etc/init.d/haproxy start
fi
exit 0
# vi check_haproxy_url.sh#通过url检测如果不成功返回非0，待达到次数后，keepalived会降权值变为backup节点
#!/bin/bash
# curl -ILhttp://localhost/member/login.htm
# curl --data"memberName=fengkan&password=22" http://localhost/member/login.htm
count=0
for (( k=0; k<2; k++ ))
do
check_code=$( curl --connect-timeout 3 -sL -w"%{http_code}\\n" http://localhost:8088/index.html -o /dev/null )
if [ "$check_code" != "200" ]; then
# count = count +1
let "count += 1"
continue
else
count=0
break
fi
done
if [ "$count" != "0" ];then
#/etc/init.d/keepalived stop
exit 1
else
exit 0
fi
# vi down.sh#维护用的脚本，不需要手动关闭keepalived
#!/bin/bash
#判断down文件是否存在，在需要维护的时候，建立一个down文件，虚拟地址会自动转移走
if [-f /etc/keepalived/down ]; then
exit 1
else
exit 0
fi
# vi vrrp.sh (ceph4)#状态发生变换，邮件提醒
#!/bin/bash
#当状态发生变换的时候，发送邮件提醒
echo "192.168.238.138 ceph4$1状态被激活，请确认HAProxy服务运行状态"|mail -s "HAProxy状态切换警告"[email protected]
# vi vrrp.sh (ceph5)#状态发生变换，邮件提醒
#!/bin/bash
echo "192.168.238.139 ceph5$1状态被激活，请确认HAProxy服务运行状态"|mail -s "HAProxy状态切换警告"[email protected]
建立完脚本后不要忘记赋予可执行的权限
#chmod +x check_haproxy.sh check_haproxy_url.sh vrrp.sh down.sh
Keepalived主配置文件
这里ceph4为master节点，ceph5为backup节点
ceph4：
vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen
}
notification_email_from [email protected]
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_haproxy_url {
script "/etc/keepalived/check_haproxy_url.sh"#查看链接是否能正常访问，不正常两次后降级，看下边的配置
interval 2# check every 2 seconds
weight -5
fall 2#失败两次后，触发weight减5操作,想有降级操作必须有
rise 2#成功两次后，恢复
}
vrrp_script chk_haproxy {
script "/etc/keepalived/check_haproxy.sh"#查看haproxy进程是否存在，不存在的话启动，无降权
interval 2#check every 2 seconds，执行的时间间隔
}
vrrp_script chk_mantaince_down {
script "/etc/keepalived/down.sh"
interval 2# check every 2 seconds
weight -5
fall 2#维护操作命令，在/etc/keepalived建立down文件开始维护
rise 2
}
vrrp_instance VI_1 {
state MASTER#这里主备不一样，注意
interface ens33#根据自己的网卡修改
virtual_router_id 50
#nopreempt
priority 101#设置优先级
advert_int 1
virtual_ipaddress {
192.168.238.200#虚拟IP地址
}
track_script {
chk_haproxy_url#与上边的执行vrrp_script脚本对应
chk_haproxy
chk_mantaince_down
}
#状态转换的时候，邮件告警
notify_backup "/etc/keepalived/vrrp.shBACKUP"
notify_master "/etc/keepalived/vrrp.shMASTER"
notify_fault"/etc/keepalived/vrrp.shFAULT"
}
Ceph5
backup节点
#这里只标出与master不一样的地方，其他同上
vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen
}
notification_email_from [email protected]
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script chk_haproxy_url {
script "/etc/keepalived/check_haproxy_url.sh"# cheaper than pidof
interval 2#check every 2 seconds
weight -5
fall 2
rise 2
}
vrrp_script chk_haproxy {
script "/etc/keepalived/check_haproxy.sh"# cheaper than pidof
interval 2#check every 2 seconds
}
vrrp_script chk_mantaince_down {
script "/etc/keepalived/down.sh"
interval 2#check every 2 seconds
weight -5
fall 2
rise 2
}
vrrp_instance VI_1 {
state BACKUP#这里为BACKUP
interface ens33
virtual_router_id 50
#nopreempt
priority 100#设置级别
advert_int 1
virtual_ipaddress {
192.168.238.200
}
track_script {
chk_haproxy_url
chk_haproxy
chk_mantaince_down
}
notify_backup "/etc/keepalived/vrrp.shBACKUP"
notify_master "/etc/keepalived/vrrp.shMASTER"
notify_fault"/etc/keepalived/vrrp.shFAULT"
}
测试：
分别启动keepalived
会看到
Ceph4日志：
# tailf /var/log/messages
Sep 21 15:09:55 ceph4 Keepalived[50677]:Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
Sep 21 15:09:55 ceph4 Keepalived[50677]:Unable to resolve default script username 'keepalived_script' - ignoring
Sep 21 15:09:55 ceph4 Keepalived[50677]:Opening file '/etc/keepalived/keepalived.conf'.
Sep 21 15:09:55 ceph4 systemd: PID file/var/run/keepalived.pid not readable (yet?) after start.
Sep 21 15:09:55 ceph4 Keepalived[50678]:Starting Healthcheck child process, pid=50679
Sep 21 15:09:55 ceph4 Keepalived[50678]:Starting VRRP child process, pid=50680
Sep 21 15:09:55 ceph4 systemd: Started LVSand VRRP High Availability Monitor.
Sep 21 15:09:55 ceph4Keepalived_healthcheckers[50679]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Registering Kernel netlink reflector
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Registering Kernel netlink command channel
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Registering gratuitous ARP shared channel
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) removing protocol VIPs.
Sep 21 15:09:55 ceph4 Keepalived_vrrp[50680]:Unsafe permissions found for script '/etc/keepalived/check_haproxy_url.sh'.
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: SECURITY VIOLATION - scripts are being executed butscript_security not enabled. There are insecure scripts.
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: Using LinkWatch kernel netlink reflector...
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP sockpool: [ifindex(2), proto(112), unicast(0),fd(10,11)]
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_mantaince_down) succeeded
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_haproxy) succeeded
Sep 21 15:09:55 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_haproxy_url) succeeded
Sep 21 15:09:56 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep21 15:09:57 ceph4 Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Entering MASTERSTATE#现在虚拟地址在主节点上
Sep 21 15:09:57 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 21 15:09:57 ceph4Keepalived_vrrp[50680]: Sending gratuitous ARP on ens33 for 192.168.238.200
测试一、现在我在ceph4（模拟不关机维护）
# touch /etc/keepalived/down#创建down文件
Sep 21 15:12:49 ceph4Keepalived_vrrp[50680]: /etc/keepalived/down.sh exited with status 1
Sep 21 15:12:51 ceph4Keepalived_vrrp[50680]: /etc/keepalived/down.sh exited with status 1
Sep 21 15:12:51 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_mantaince_down) failed
Sep 21 15:12:51 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Changing effective priority from101 to 96
Sep 21 15:12:52 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Received advert with higherpriority 100, ours 96
Sep 21 15:12:52 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1)Entering BACKUP STATE#在创建down文件后，weight降级了，变为了BACKUP节点
Sep 21 15:12:52 ceph4 Keepalived_vrrp[50680]:VRRP_Instance(VI_1) removing protocol VIPs.
邮箱收到信息如下：
Ceph4变为backup
Ceph5变为激活master
说明测试成功，地址已经漂移到ceph5，ceph4可以维护了
将down文件删掉后
# rmdown
rm:remove regular empty file ‘down’? y
ceph4
Sep 21 15:17:18 ceph4Keepalived_vrrp[50680]: VRRP_Script(chk_mantaince_down) succeeded
Sep 21 15:17:18 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Changing effective priority from 96to 101
Sep 21 15:17:18 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) forcing a new MASTER election
Sep 21 15:17:19 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep21 15:17:20 ceph4 Keepalived_vrrp[50680]: VRRP_Instance(VI_1) Entering MASTERSTATE#变为主的状态
Sep 21 15:17:20 ceph4Keepalived_vrrp[50680]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 21 15:17:20 ceph4Keepalived_vrrp[50680]: Sending gratuitous ARP on ens33 for 192.168.238.200
Ceph5：
Sep 21 15:17:18 ceph5Keepalived_vrrp[11531]: VRRP_Instance(VI_1) Received advert with higherpriority 101, ours 100
Sep 21 15:17:18 ceph5Keepalived_vrrp[11531]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep21 15:17:18 ceph5 Keepalived_vrrp[11531]: VRRP_Instance(VI_1) removing protocolVIPs.#地址已经漂移走了,回到了ceph4
