haproxy动态增减主机与keepalived高级应用

一：本文将详细介绍haproxy的配置使用以及高级功能的使用，比如通过haproxy进行动态添加删除负载集群中的后端web服务器的指定主机，另外将详细介绍keepalived的详细配置方法、配置实例及实现方式，具体如下：

1.1：通过haproxy动态增减web集群中的主机，打开haproxy的主机在线维护功能，主要原理是haproxy监听本地socket文件，mysql写localhost会查抄默认路径为/var/lib/mysql/mysql.socket文件，但是使用IP地址是使用的TCP协议访问，就不需要查找socket文件了，所以使用localhost连接本机MySQL和写127.0.0.1连接本机MySQL的实现方式是不一样的，haproxy配置socket文件路径：

1.1.1：haproxy配置文件添加socket配置信息如下：

[root@localhost ~]# cat /etc/haproxy/haproxy.cfg 
global
maxconn 100000 chroot /usr/local/haproxy uid 99 gid 99 daemon nbproc 1 pidfile /usr/local/haproxy/logs/haproxy.pid log 127.0.0.1 local0 info stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin #设置socket文件路径，权限为600级别为admin，还有其他级别，但是动态添加主机需要admin权限

1.1.2：创建保存socket文件的目录：

# mkdir  /var/lib/haproxy

1.1.3：重启haproxy服务：

[root@localhost ~]# /etc/init.d/haproxy  restart
Restarting haproxy (via systemctl): [ OK ]

1.1.4：验证haproxy的socket文件：

[root@localhost ~]# ll /var/lib/haproxy/
total 0 srw------- 1 root root 0 Dec 31 21:02 haproxy.socket

1.2：安装socat命令，socat是和socket通信的命令：

1.2.1：安装命令：

[root@localhost ~]# yum install socat

1.2.2：命令使用帮助：

[root@localhost ~]# echo "help" | socat stdio /var/lib/haproxy/haproxy.socket 
Unknown command. Please enter one of the following commands only : help : this message prompt : toggle interactive mode with prompt quit : disconnect disable agent : disable agent checks (use 'set server' instead) disable health : disable health checks (use 'set server' instead) disable server : disable a server for maintenance (use 'set server' instead) enable agent : enable agent checks (use 'set server' instead) enable health : enable health checks (use 'set server' instead) enable server : enable a disabled server (use 'set server' instead) set maxconn server : change a server's maxconn setting set server : change a server's state, weight or address get weight : report a server's current weight set weight : change a server's weight (deprecated) disable frontend : temporarily disable specific frontend enable frontend : re-enable specific frontend set maxconn frontend : change a frontend's maxconn setting show servers state [id]: dump volatile server information (for backend ) show backend : list backends in the current running config shutdown frontend : stop a specific frontend clear table : remove an entry from a table set table [id] : update or create a table entry's data show table [id]: report table usage stats or dump this table's contents show errors : report last request and response errors for each proxy clear counters : clear max statistics counters (add 'all' for all counters) show info : report information about the running process show stat : report counters for each proxy and server show sess [id] : report the list of current sessions or dump this session shutdown session : kill a specific session shutdown sessions server : kill sessions on a server show pools : report information about the memory pools usage add acl : add acl entry clear acl  : clear the content of this acl del acl : delete acl entry get acl : report the patterns matching a sample for an ACL show acl [id] : report available acls or dump an acl's contents add map : add map entry clear map <id> : clear the content of this map del map : delete map entry get map : report the keys and values matching a sample for a map set map : modify map entry show map [id] : report available maps or dump a map's contents show stat resolvers [id]: dumps counters from all resolvers section and associated name servers set maxconn global : change the per-process maxconn setting set rate-limit : change a rate limiting value set timeout : change a timeout setting show env [var] : dump environment variables known to the process show tls-keys [id|*]: show tls keys references or dump tls ticket keys when id specifie

1.2.3：查看haproxy的详细信息：

[root@localhost ~]# echo "show info" | socat stdio /var/lib/haproxy/haproxy.socket 
Name: HAProxy Version: 1.7.1 Release_date: 2016/12/13 Nbproc: 1 Process_num: 1 Pid: 20077 Uptime: 0d 0h04m49s Uptime_sec: 289 Memmax_MB: 0 PoolAlloc_MB: 0 PoolUsed_MB: 0 PoolFailed: 0 Ulimit-n: 200034 Maxsock: 200034 Maxconn: 100000 Hard_maxconn: 100000 CurrConns: 0 CumConns: 3 CumReq: 3 MaxSslConns: 0 CurrSslConns: 0 CumSslConns: 0 Maxpipes: 0 PipesUsed: 0 PipesFree: 0 ConnRate: 0 ConnRateLimit: 0 MaxConnRate: 0 SessRate: 0 SessRateLimit: 0 MaxSessRate: 0 SslRate: 0 SslRateLimit: 0 MaxSslRate: 0 SslFrontendKeyRate: 0 SslFrontendMaxKeyRate: 0 SslFrontendSessionReuse_pct: 0 SslBackendKeyRate: 0 SslBackendMaxKeyRate: 0 SslCacheLookups: 0 SslCacheMisses: 0 CompressBpsIn: 0 CompressBpsOut: 0 CompressBpsRateLim: 0 ZlibMemUsage: 0 MaxZlibMemUsage: 0 Tasks: 7 Run_queue: 1 Idle_pct: 100 node: localhost

1.3：关闭一个后端服务器中的web服务器并在管理界面进行验证：

1.3.1：动态关闭一台web服务器：

[root@localhost ~]# echo "disable server web_port/web1" | socat stdio /var/lib/haproxy/haproxy.socket  #关闭指定backend下的某台web服务器

1.3.2：在haproxy的管理端进行验证该web服务器是否从负载中下线：

1.3.3：动态开启服务器：

[root@localhost ~]# echo "enable  server web_port/web1" | socat stdio /var/lib/haproxy/haproxy.socket

1.3.4：验证服务器是否开启：

二：Keepalived 应用：

官方下载地址：http://www.keepalived.org/download.html

Keepalived权威指南：Keepalived权威指南
2.1：keepalived是VRRP协议在linux系统上的完美实现的解决方案，详细如下：

    虚拟路由冗余协议(Virtual Router Redundancy Protocol，简称VRRP)是由IETF提出的解决局域网中配置静态网关出现单点失效现象的路由协议，1998年已推出正式的RFC2338协议标准。VRRP广泛应用在边缘网络中，它的设计目标是支持特定情况下IP数据流量失败转移不会引起混乱，允许主机使用单路由器，以及及时在实际第一跳路由器使用失败的情形下仍能够维护路由器间的连通性。VRRP是一种选择协议，它可以把一个虚拟路由器的责任动态分配到局域网上的 VRRP 路由器中的一台。控制虚拟路由器 IP 地址的 VRRP 路由器称为主路由器，它负责转发数据包到这些虚拟 IP 地址。[1]  一旦主路由器不可用，这种选择过程就提供了动态的故障转移机制，这就允许虚拟路由器的 IP 地址可以作为终端主机的默认第一跳路由器。是一种LAN接入设备备份协议。一个局域网络内的所有主机都设置缺省网关，这样主机发出的目的地址不在本网段的报文将被通过缺省网关发往三层交换机，从而实现了主机和外部网络的通信。
VRRP是一种路由容错协议，也可以叫做备份路由协议。一个局域网络内的所有主机都设置缺省路由，当网内主机发出的目的地址不在本网段时，报文将被通过缺省路由发往外部路由器，从而实现了主机与外部网络的通信。当缺省路由器down掉（即端口关闭）之后，内部主机将无法与外部通信，如果路由器设置了VRRP时，那么这时，虚拟路由将启用备份路由器，从而实现全网通信。
脑裂是backup收不到master的心跳组播包，可以使用网线直连做心跳。

2.2：部署两台服务器，各安装haproxy和keepalived，实现负载高可用：

2.2.1：haproxy安装方式在上一篇博客有详细介绍：

2.2.2：编译安装keepalived：

 

# tar xvf keepalived-1.2.24.tar.gz
# cd keepalived-1.2.24
# ./configure  --prefix=/usr/local/keepalived-1.2.24 # make && make install # ln -sv /usr/local/keepalived-1.2.24/ /usr/local/keepalived # cp keepalived/etc/init.d/keepalived.sysconfig /etc/sysconfig/keepalived # cp keepalived/etc/init.d/keepalived.init /etc/init.d/keepalive # chmod a+x /etc/init.d/keepalived # mkdir /etc/keepalived  # ln -sv /usr/local/keepalived/sbin/keepalived /usr/bin/ #不做软连接报错，提示/bin/bash/keepakived找不到

2.2.3：server1 编辑keepalived配置文件：

! Configuration File for keepalived global_defs { notification_email {  #指定keepalived在发生事件，比如切换IP的时候需要发生邮件的地址，可以有多个，一行一个即可  acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc  #指定smtp服务器设置  smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL  #运行keepalived的标示，可以写主机名等信息  } vrrp_instance IP1 {  #监听一组VIP  state BACKUP #指定初始状态为backup即备份状态 interface br0 #监听的本地网卡名称,此处是一块桥接网卡， virtual_router_id 10  #路由id，多个网卡环境下，id一定不能一样，范围为10-255  priority 90 #优先级，高的为master  advert_int 1  #进行通告  authentication { #配置认证机制  auth_type PASS #认证类型为密码  auth_pass 5555 #认证密码 } virtual_ipaddress { 192.168.10.100 } } vrrp_instance IP2 { state MASTER #第二组IP初始状态是master interface br0 virtual_router_id 20 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.10.101 #具体的虚拟IP地址，两个keepalived设置要一样，否则服务不生效 } }

2.3.4：server 2的keepalived配置：

# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance IP1 { state MASTER #服务器1的master interface eth1 virtual_router_id 10 priority 150 advert_int 1 authentication { auth_type PASS auth_pass 5555 } virtual_ipaddress { 192.168.10.100 } } vrrp_instance IP2 { state BACKUP #是服务器1的备份 interface eth1 virtual_router_id 20 priority 50 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.10.101 } }

2.4：关闭一个keepalived服务，在日志记录中验证IP地址是否会迁移到backup服务器：

2.4.1：在server 1关闭keepalived：

[root@localhost keepalived-1.2.24]# /etc/init.d/keepalived stop Stopping keepalived (via systemctl): [ OK 

2.4.2：在server 2查看日志：

2.4.3：验证server 2的IP地址：

2.4.4：开启server 1的keepalived：

[root@localhost keepalived-1.2.24]# /etc/init.d/keepalived start Starting keepalived (via systemctl): [ OK ]

2.4.5：验证server 2的日志：

2.4.6：验证server 2的IP地址：

2.4.7：验证server 1的ip地址：

2.4.8：使用tcpdump抓取心跳检测的数据包：

2.5：关于keepalived的不抢占：

抢占发生在Master和Backup环境中Master 出现问题导致虚拟IP漂移到了Backup，但是当Master恢复之后IP地址会再次漂移到Master上，并且反反复复出现这样的问题，从而导致出现出现频繁网络切换，而可能引起未知的网络问题，因此可以配置不抢占，当IP漂移到了Backup之后不再往回迁IP，需要手动将IP配置回去才可以，具体如下：

2.5.1：Server1的不抢占的详细配置：

[root@localhost keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf  ! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance IP1 { state BACKUP interface br0 virtual_router_id 10 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 5555 } virtual_ipaddress { 192.168.10.100 } } vrrp_instance IP2 { state BACKUP #不抢占需要配置在BACKUP组内，并且优先级高于备份服务器的keepalived组内 interface br0 virtual_router_id 20 nopreempt #声明不抢占，即当IP飘走之后不再根据优先级抢占IP，但是当对方服务出现问题依然可以绑定IP到本地 priority 100 #配置在高优先级的keepalived组内，低优先级的就不需要配置抢占了 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.10.101 } }

2.5.2：Serevr 2不抢占的详细配置：

[root@node10 ~]# cat  /etc/keepalived/keepalived.conf 
! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance IP1 { state BACKUP #状态必须是BACKUP interface eth1 virtual_router_id 10 priority 150 nopreempt #不抢占，优先级比Server 1的同组高 advert_int 1 authentication { auth_type PASS auth_pass 5555 } virtual_ipaddress { 192.168.10.100 } } vrrp_instance IP2 { state BACKUP #状态为BACKUP interface eth1 virtual_router_id 20 priority 50 #比Server 1的的同组低，就不需要配置不抢占了 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.10.101 } }

2.5.3：查看IP现在在那个服务器，将keepalived服务关闭在开启，验证IP地址是否还会配置在本地：

[root@localhost keepalived-1.2.24]# ip addr | grep 192 inet 192.168.10.128/24 brd 192.168.10.255 scope global br0 inet 192.168.10.101/32 scope global br0 inet 192.168.10.100/32 scope global br0 inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 [root@localhost keepalived-1.2.24]# /etc/init.d/keepalived stop Stopping keepalived (via systemctl): [ OK ] [root@localhost keepalived-1.2.24]#  [root@localhost keepalived-1.2.24]# ip addr | grep 192 inet 192.168.10.128/24 brd 192.168.10.255 scope global br0 inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 [root@localhost keepalived-1.2.24]# /etc/init.d/keepalived start Starting keepalived (via systemctl): [ OK ] [root@localhost keepalived-1.2.24]# ip addr | grep 192 inet 192.168.10.128/24 brd 192.168.10.255 scope global br0 inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

2.5.4：验证server 2的IP地址：

[root@node10 ~]# ip addr | grep 192
    inet 192.168.10.129/24 brd 192.168.10.255 scope global eth1 inet 192.168.10.100/32 scope global eth1 inet 192.168.10.101/32 scope global eth1 inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

注：由此可见，在非抢占模式下，IP地址发送变动后不会再向优先级高的服务器进行漂移，减少了因为IP变动而引起的网络故障。