HAProxy的高级配置选项-配置haproxy支持https协议及服务器动态上下线
作者:尹正杰
版权声明:原创作品,谢绝转载!否则将追究法律责任。
一.证书制作
1>.创建私钥
[[email protected] ~]# mkdir -pv /yinzhengjie/softwares/haproxy/certs mkdir: created directory ‘/yinzhengjie/softwares/haproxy/certs’ [[email protected] ~]# [[email protected] ~]# cd /yinzhengjie/softwares/haproxy/certs/ [[email protected] /yinzhengjie/softwares/haproxy/certs]# [[email protected] /yinzhengjie/softwares/haproxy/certs]# openssl genrsa -out haproxy.key 2048 Generating RSA private key, 2048 bit long modulus ...........................+++ ......................................................................+++ e is 65537 (0x10001) [[email protected] /yinzhengjie/softwares/haproxy/certs]# [[email protected] /yinzhengjie/softwares/haproxy/certs]# ll total 4 -rw-r--r-- 1 root root 1675 Jan 7 07:18 haproxy.key [[email protected] /yinzhengjie/softwares/haproxy/certs]#
2>.基于私钥创建一个crt文件
[[email protected] /yinzhengjie/softwares/haproxy/certs]# ll total 4 -rw-r--r-- 1 root root 1675 Jan 7 07:18 haproxy.key [[email protected] /yinzhengjie/softwares/haproxy/certs]# [[email protected] /yinzhengjie/softwares/haproxy/certs]# openssl req -new -x509 -key haproxy.key -out haproxy.crt -subj "/CN=node102.yinzhengjie.org.cn" [[email protected] /yinzhengjie/softwares/haproxy/certs]# [[email protected] /yinzhengjie/softwares/haproxy/certs]# ll total 8 -rw-r--r-- 1 root root 1139 Jan 7 07:21 haproxy.crt -rw-r--r-- 1 root root 1675 Jan 7 07:18 haproxy.key [[email protected] /yinzhengjie/softwares/haproxy/certs]# [[email protected] /yinzhengjie/softwares/haproxy/certs]#
3>.生成一个haproxy使用的证书文件
[[email protected] /yinzhengjie/softwares/haproxy/certs]# ll total 8 -rw-r--r-- 1 root root 1139 Jan 7 07:21 haproxy.crt -rw-r--r-- 1 root root 1675 Jan 7 07:18 haproxy.key [[email protected] /yinzhengjie/softwares/haproxy/certs]# [[email protected] /yinzhengjie/softwares/haproxy/certs]# cat haproxy.key haproxy.crt > haproxy.pem #生成证书文件 [[email protected] /yinzhengjie/softwares/haproxy/certs]# [[email protected] /yinzhengjie/softwares/haproxy/certs]# ll total 12 -rw-r--r-- 1 root root 1139 Jan 7 07:21 haproxy.crt -rw-r--r-- 1 root root 1675 Jan 7 07:18 haproxy.key -rw-r--r-- 1 root root 2814 Jan 7 07:23 haproxy.pem [[email protected] /yinzhengjie/softwares/haproxy/certs]#
[[email protected] /yinzhengjie/softwares/haproxy/certs]# openssl x509 -in haproxy.pem -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: c1:7d:0d:33:31:a0:2a:86 Signature Algorithm: sha256WithRSAEncryption Issuer: CN=node102.yinzhengjie.org.cn Validity Not Before: Jan 6 23:21:42 2020 GMT Not After : Feb 5 23:21:42 2020 GMT Subject: CN=node102.yinzhengjie.org.cn Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:b5:8d:25:2d:1c:22:c5:01:c4:47:8b:87:6b:3a: f9:34:d5:db:0b:3a:34:10:42:a6:33:24:cc:e7:3b: 26:01:18:ee:2d:e3:e4:24:c9:8a:12:aa:1c:8e:fb: 38:60:bc:1a:0b:c5:85:48:ea:36:83:86:d3:50:6d: 85:3c:14:43:10:9e:87:d0:40:54:c5:58:15:4d:a6: 68:1f:c3:aa:1b:fb:9c:d6:d4:3e:33:8a:d4:d6:00: d4:e2:a4:22:e8:06:77:35:80:40:48:83:3c:1c:12: 1e:33:d3:97:64:c8:37:06:d2:1d:c9:c1:a0:f4:c9: d2:56:c7:43:a6:9f:79:a4:e1:51:23:d7:90:20:bc: 30:ee:cd:ac:10:fa:0b:db:ea:a7:65:4b:fb:24:fb: 97:4b:2a:6f:7d:52:04:1e:ea:74:df:8c:53:09:ca: 38:61:a7:2d:e8:33:c7:76:5f:37:aa:d3:df:f6:b4: ca:76:42:24:21:c2:40:1d:d1:9f:2d:9b:01:62:b4: 2d:55:4f:71:ae:8b:29:3c:ab:fb:47:1b:5c:8f:67: c0:80:71:d3:d5:d7:0a:b5:9f:51:5a:56:c3:de:70: a5:4a:fa:c7:69:65:47:22:6c:96:ee:57:1a:4b:f1: ef:5f:09:1b:e6:15:ce:4a:14:06:8d:4d:f3:d8:a5: e8:c3 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: F2:B1:1F:87:C5:37:3C:F6:00:A6:F6:06:59:05:D3:48:58:BB:F3:8C X509v3 Authority Key Identifier: keyid:F2:B1:1F:87:C5:37:3C:F6:00:A6:F6:06:59:05:D3:48:58:BB:F3:8C X509v3 Basic Constraints: CA:TRUE Signature Algorithm: sha256WithRSAEncryption 1d:5b:c4:a5:ef:f4:41:f1:06:40:67:a1:d7:9a:20:4b:5b:3e: 1b:d7:8c:84:39:4f:ce:62:5f:e9:48:b7:3b:80:12:de:00:8e: eb:13:83:70:28:9c:2e:6f:0f:9c:2d:92:0d:f7:d4:7b:cc:e3: eb:67:c4:48:2a:f0:ad:57:f9:51:28:75:6b:86:12:0c:28:8b: ba:45:55:df:95:ed:68:b6:27:47:71:b6:44:11:9d:29:f5:b5: 68:b7:db:30:76:a2:79:bc:cb:60:9b:68:e3:5e:b5:00:da:c5: c5:4d:ff:f9:9d:fe:28:66:00:b2:b2:d7:36:ef:05:15:d6:26: 44:4a:d4:e4:1a:06:9b:f1:42:f1:f5:b7:32:98:5a:78:70:b9: f2:26:45:8e:db:a5:3b:5c:9b:c4:35:54:63:e7:18:d6:55:4c: 1b:47:0b:b8:e3:99:b3:b0:e9:d1:50:f5:50:b8:8c:3d:2f:d3: 7b:54:57:52:6b:4d:d1:07:31:96:cc:3f:72:67:0b:db:de:d8: e8:14:f2:a3:c4:ff:41:24:90:12:8d:0c:45:64:cd:2b:c1:ce: ab:f5:c6:b4:e7:36:bf:f4:5e:d8:7a:36:94:a8:9d:99:60:2f: d7:04:f8:58:e9:9f:9d:25:92:c6:ab:c0:c2:30:04:91:92:17: 81:54:9b:ff [[email protected] /yinzhengjie/softwares/haproxy/certs]# [[email protected] /yinzhengjie/softwares/haproxy/certs]#
二.配置haproxy支持https协议案例
1>.编辑haproxy的启动脚本,让其支持从多个路径读取配置文件
[[email protected] ~]# haproxy --help HA-Proxy version 1.8.20 2019/04/25 Copyright 2000-2019 Willy TarreauUsage : haproxy [-f ]* [ -vdVD ] [ -n ] [ -N ] [ -p ] [ -m ] [ -C <dir> ] [-- *] -v displays version ; -vv shows known build options. -d enters debug mode ; -db only disables background mode. -dM[<byte>] poisons memory with <byte> (defaults to 0x50) -V enters verbose mode (disables quiet mode) -D goes daemon ; -C changes to <dir> before loading files. -W master-worker mode. -Ws master-worker mode with systemd notify support. -q quiet mode : don't display messages -c check mode : only check config files and exit -n sets the maximum total # of connections (2000) -m limits the usable amount of memory (in MB) -N sets the default, per-proxy maximum # of connections (2000) -L set local peer name (default to hostname) -p writes pids of all children to this file -de disables epoll() usage even when available -dp disables poll() usage even when available -dS disables splice usage (broken on old kernels) -dR disables SO_REUSEPORT usage -dr ignores server address resolution failures -dV disables SSL verify on servers side -sf/-st [pid ]* finishes/terminates old pids. -x get listening sockets from a unix socket [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# cat /usr/lib/systemd/system/haproxy.service #这是咱们之前的配置文件 [Unit] Description=Yinzhengjie's HAProxyLoad Balancer After=syslog.target network.target [Service] ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q ExecStart=/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /yinzhengjie/softwares/haproxy/haproxy.pid ExecReload=/bin/kill -USR2 $MAINPID [Install] WantedBy=multi-user.target [[email protected] ~]# [[email protected] ~]# [[email protected] ~]# vim /usr/lib/systemd/system/haproxy.service [[email protected] ~]# [[email protected] ~]# cat /usr/lib/systemd/system/haproxy.service [Unit] Description=Yinzhengjie's HAProxyLoad Balancer After=syslog.target network.target [Service] ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -c -q ExecStart=/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /yinzhengjie/softwares/haproxy/haproxy.pid ExecReload=/bin/kill -USR2 $MAINPID [Install] WantedBy=multi-user.target [[email protected] ~]# [[email protected] ~]# systemctl daemon-reload #使得配置文件生效。 [[email protected] ~]#
2>.创建haproxy的子配置文件
[[email protected] ~]# ll /etc/haproxy/ total 12 -rw-r--r-- 1 root root 1822 Jan 7 07:47 haproxy.cfg -rw-r--r-- 1 root root 1317 Jan 4 10:29 haproxy.cfg-2020-01-04 -rw-r--r-- 1 root root 1697 Jan 5 06:32 haproxy.cfg-2020-01-05 [[email protected] ~]# [[email protected] ~]# [[email protected] ~]# mkdir -pv /etc/haproxy/conf.d mkdir: created directory ‘/etc/haproxy/conf.d’ [[email protected] ~]# [[email protected] ~]# ll /etc/haproxy/ total 12 drwxr-xr-x 2 root root 44 Jan 7 07:51 conf.d -rw-r--r-- 1 root root 915 Jan 7 07:51 haproxy.cfg -rw-r--r-- 1 root root 1317 Jan 4 10:29 haproxy.cfg-2020-01-04 -rw-r--r-- 1 root root 1697 Jan 5 06:32 haproxy.cfg-2020-01-05 [[email protected] ~]#
3>.编辑haproxy的主配置文件
[[email protected] ~]# cat /etc/haproxy/haproxy.cfg global maxconn 100000 chroot /yinzhengjie/softwares/haproxy stats socket /yinzhengjie/softwares/haproxy/haproxy.sock mode 600 level admin user haproxy group haproxy daemon nbproc 2 cpu-map 1 0 cpu-map 2 1 nbthread 2 pidfile /yinzhengjie/softwares/haproxy/haproxy.pid log 127.0.0.1 local5 info defaults option http-keep-alive option forwardfor option redispatch option abortonclose maxconn 100000 mode http timeout connect 300000ms timeout client 300000ms timeout server 300000ms errorloc 503 http://node107.yinzhengjie.org.cn/monitor/503.html listen status_page bind 172.30.1.102:8888 stats enable stats uri /haproxy-status stats auth admin:yinzhengjie stats realm "Welcome to the haproxy load balancer status page of YinZhengjie" stats hide-version stats admin if TRUE stats refresh 5s [[email protected] ~]# [[email protected] ~]#
4>.编辑haproxy的子配置文件
[[email protected] ~]# cat /etc/haproxy/conf.d/node102_yinzhengjie_org_cn.cfg listen WEB_PROT_80 bind 172.30.1.102:80 mode http #将http的请求重定向为https请求 redirect scheme https if !{ ssl_fc } balance leastconn server web01 172.30.1.106:80 check server web02 172.30.1.107:80 check server web03 172.30.1.108:80 check backup listen WEB_PROT_443 bind 172.30.1.102:443 ssl crt /yinzhengjie/softwares/haproxy/certs/haproxy.pem mode http #将客户端请求的源端口转发给后端服务器,以便于后端web服务器有相应的记录日志 http-request set-header X-Forwarded-Port %[dst_port] #将客户端请求的协议转发给后端服务器,一百年与后端的web服务器有相应的记录日志 http-request add-header X-Forwarded-Proto https if { ssl_fc } balance leastconn server web01 172.30.1.106:80 check server web02 172.30.1.107:80 check server web03 172.30.1.108:80 check backup [[email protected] ~]# [[email protected] ~]#
5>.启动haproxy服务并查看状态页
[[email protected] ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 *:22 *:* LISTEN 0 128 :::22 :::* [[email protected] ~]# [[email protected] ~]# systemctl start haproxy [[email protected] ~]# [[email protected] ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 172.30.1.102:80 *:* LISTEN 0 128 *:22 *:* LISTEN 0 128 172.30.1.102:8888 *:* LISTEN 0 128 172.30.1.102:443 *:* LISTEN 0 128 :::22 :::* [[email protected] ~]# [[email protected] ~]#
6>.浏览器访问"http:node102.yinzhengjie.org.cn"
三.配置haproxy服务器动态上下线案例实战
1>.查看服务器的cpu核心数
[[email protected] ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 8 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 158 Model name: Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz Stepping: 10 CPU MHz: 2207.998 BogoMIPS: 4415.99 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 9216K NUMA node0 CPU(s): 0-7 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase avx2 invpcid rdseed clflushopt flush_l1d[[email protected] ~]#
2>.编辑haproxy的主配置文件
[[email protected] ~]# lscpu | grep "CPU(s):" CPU(s): 8 NUMA node0 CPU(s): 0-7 [[email protected] ~]# [[email protected] ~]# cat haproxy_sock.sh #编写快速生成socket文件的脚本 #咱们这里有多少个核心就生成多少个数字 for i in `seq 1 8` do echo "stats socket /yinzhengjie/softwares/haproxy/haproxy${i}.sock mode 600 level admin process $i" done [[email protected] ~]# [[email protected] ~]# sh haproxy_sock.sh stats socket /yinzhengjie/softwares/haproxy/haproxy1.sock mode 600 level admin process 1 stats socket /yinzhengjie/softwares/haproxy/haproxy2.sock mode 600 level admin process 2 stats socket /yinzhengjie/softwares/haproxy/haproxy3.sock mode 600 level admin process 3 stats socket /yinzhengjie/softwares/haproxy/haproxy4.sock mode 600 level admin process 4 stats socket /yinzhengjie/softwares/haproxy/haproxy5.sock mode 600 level admin process 5 stats socket /yinzhengjie/softwares/haproxy/haproxy6.sock mode 600 level admin process 6 stats socket /yinzhengjie/softwares/haproxy/haproxy7.sock mode 600 level admin process 7 stats socket /yinzhengjie/softwares/haproxy/haproxy8.sock mode 600 level admin process 8 [[email protected] ~]#
[[email protected] ~]# cat /etc/haproxy/haproxy.cfg global maxconn 100000 chroot /yinzhengjie/softwares/haproxy user haproxy group haproxy daemon #开启8个进程 nbproc 8 cpu-map 1 0 cpu-map 2 1 nbthread 2 #由于上面开启了多进程,而每个套接字同事只能对一个套接字发送指令,因此在模拟服务器动态上下线时, #在状态页面我们会发现有的进程是记录某个节点是下线状态的,某个节点是关闭状态的。因此,为了解决这 #个问题,我们需要手动配置多个套接字文件,即上面开启了多少个线程,咱们这里就得写多少个套接字 stats socket /yinzhengjie/softwares/haproxy/haproxy1.sock mode 600 level admin process 1 stats socket /yinzhengjie/softwares/haproxy/haproxy2.sock mode 600 level admin process 2 stats socket /yinzhengjie/softwares/haproxy/haproxy3.sock mode 600 level admin process 3 stats socket /yinzhengjie/softwares/haproxy/haproxy4.sock mode 600 level admin process 4 stats socket /yinzhengjie/softwares/haproxy/haproxy5.sock mode 600 level admin process 5 stats socket /yinzhengjie/softwares/haproxy/haproxy6.sock mode 600 level admin process 6 stats socket /yinzhengjie/softwares/haproxy/haproxy7.sock mode 600 level admin process 7 stats socket /yinzhengjie/softwares/haproxy/haproxy8.sock mode 600 level admin process 8 pidfile /yinzhengjie/softwares/haproxy/haproxy.pid log 127.0.0.1 local5 info defaults option http-keep-alive option forwardfor option redispatch option abortonclose maxconn 100000 mode http timeout connect 300000ms timeout client 300000ms timeout server 300000ms errorloc 503 http://node107.yinzhengjie.org.cn/monitor/503.html listen status_page bind 172.30.1.102:8888 stats enable stats uri /haproxy-status stats auth admin:yinzhengjie stats realm "Welcome to the haproxy load balancer status page of YinZhengjie" stats hide-version stats admin if TRUE stats refresh 5s [[email protected] ~]#
3>.编辑haproxy的子配置文件
[[email protected] ~]# cat /etc/haproxy/conf.d/node102_yinzhengjie_org_cn.cfg listen WEB_PROT_80 bind 172.30.1.102:80 mode http redirect scheme https if !{ ssl_fc } balance leastconn server web01 172.30.1.106:80 check server web02 172.30.1.107:80 check server web03 172.30.1.108:80 check backup listen WEB_PROT_443 bind 172.30.1.102:443 ssl crt /yinzhengjie/softwares/haproxy/certs/haproxy.pem mode http http-request set-header X-Forwarded-Port %[dst_port] http-request add-header X-Forwarded-Proto https if { ssl_fc } balance leastconn #咱们的后端web服务器名称也可以写IP地址哟,为了自动化运维管理传参方便,其实我个人还是比较推荐写IP地址的 server 172.30.1.106 172.30.1.106:80 check server 172.30.1.107 172.30.1.107:80 check server 172.30.1.108 172.30.1.108:80 check backup [[email protected] ~]# [[email protected] ~]# systemctl restart haproxy #别忘记重启haproxy使得配置文件生效哟~ [[email protected] ~]#
4>.使用socat工具使得haproxy的后端服务器("172.30.1.106")动态上下线
[[email protected] ~]# yum install socat Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile epel/x86_64/metalink | 9.6 kB 00:00:00 * base: mirrors.aliyun.com * epel: mirrors.yun-idc.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com base | 3.6 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 Package socat-1.7.3.2-2.el7.x86_64 already installed and latest version Nothing to do [[email protected] ~]#
[[email protected] ~]# echo "show info" | socat stdio /yinzhengjie/softwares/haproxy/haproxy1.sock #通过套接字查看haproxy的状态信息 Name: HAProxy Version: 1.8.20 Release_date: 2019/04/25 Nbthread: 2 Nbproc: 8 Process_num: 1 Pid: 31238 Uptime: 0d 0h05m27s Uptime_sec: 327 Memmax_MB: 0 PoolAlloc_MB: 0 PoolUsed_MB: 0 PoolFailed: 0 Ulimit-n: 200115 Maxsock: 200115 Maxconn: 100000 Hard_maxconn: 100000 CurrConns: 0 CumConns: 3 CumReq: 5 MaxSslConns: 0 CurrSslConns: 0 CumSslConns: 0 Maxpipes: 0 PipesUsed: 0 PipesFree: 0 ConnRate: 0 ConnRateLimit: 0 MaxConnRate: 1 SessRate: 0 SessRateLimit: 0 MaxSessRate: 1 SslRate: 0 SslRateLimit: 0 MaxSslRate: 0 SslFrontendKeyRate: 0 SslFrontendMaxKeyRate: 0 SslFrontendSessionReuse_pct: 0 SslBackendKeyRate: 0 SslBackendMaxKeyRate: 0 SslCacheLookups: 0 SslCacheMisses: 0 CompressBpsIn: 0 CompressBpsOut: 0 CompressBpsRateLim: 0 ZlibMemUsage: 0 MaxZlibMemUsage: 0 Tasks: 12 Run_queue: 1 Idle_pct: 100 node: node102.yinzhengjie.org.cn Stopping: 0 Jobs: 12 Listeners: 11 [[email protected] ~]# [[email protected] ~]#
[[email protected] ~]# echo "disable server WEB_PROT_443/172.30.1.106" | socat stdio /yinzhengjie/softwares/haproxy/haproxy1.sock 执行后如下图所示。 [[email protected] ~]#
如上图所示,目前只有一个进程标记"172.30.1.106"节点处于down状态。其它七个进程都标记"172.30.1.106"为正常状态,如下图所示。
解决方案: [[email protected] ~]# for i in `seq 1 8`;do echo "disable server WEB_PROT_443/172.30.1.106" | socat stdio /yinzhengjie/softwares/haproxy/haproxy${i}.sock;done