7.0下通过CARP+PF+relayd+PFSYNC实现健康检查等功能群集[转]

7.0下通过CARP+PF+relayd+PFSYNC实现健康检查等功能群集

Submitted by sysv on 2008, August 19, 3:32 PM. Cluster

在<>基础上，通过relayd来实现以下目标:

1. 对服务器池的健康检查；
2. 实现3层和7层的load balance;


一、安装relayd
1. 安装前需要确认内核配置文档中加入以下几个参数:

1. options      INET6            # IPv6 communications protocols (relayd默认对ipv6进行了支持)  
2. device       tun      # Packet tunnel.(relayd在实现上需要利用到tunnel, 必须加上)  
3.   
4. ### pf support ###  
5. device carp  
6. device pf  
7. device pflog  
8. device pfsync  
9.   
10. options          ALTQ  
11. options          ALTQ_CBQ  
12. options          ALTQ_RED  
13. options          ALTQ_RIO  
14. options          ALTQ_HFSC  
15. options          ALTQ_PRIQ  
16. options          ALTQ_NOPCC  

如重新配置了以上内核参数请重新编译内核，并重新升效

2. 开始安装relayd. 除自己DIY源代码, Makefile基本不必要改了。
# cd /usr/ports/net/relayd
# make install clean

二、配置rc.conf(加入的相关项)
# vim /etc/rc.conf

代码

1. gateway_enable="YES"  
2. defaultrouter="192.168.1.1"  
3. hostname="master.cluster.org"  
4. cloned_interfaces="carp0 carp1"  
5.   
6. # External Public Interface (for the secondary firewall use a different public ip.)  
7. ifconfig_em0="inet 192.168.1.52 netmask 255.255.255.0"  
8. # External Public Carp Interface  
9. #ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24"  
10. ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24 advskew 10"  
11.   
12. # Internal Interface (for the secondary firewall change the ip address to 192.168.10.11)  
13. ifconfig_em1="inet 192.168.10.10 netmask 255.255.255.0"  
14. # Internal Carp Interface
    # vhid应为唯一值，不可与carp0相同。否则会出问题
    
15. ifconfig_carp1="vhid 10 pass 22222 192.168.10.100/24 advskew 10"  
16.   
17. # Heartbeat Interface (for the secondary firewall, change the ip address to 10.10.10.251)  
18. ifconfig_vr0="10.10.10.250 netmask 255.255.255.0"  
19.   
20. # PFSync Interface  
21. #ifconfig_pfsync0="up syncdev vr0"  
22.   
23. pf_enable="YES"  
24. pf_rules="/etc/pf.conf"  
25. pf_flags=""  
26. pflog_enable="YES"  
27. pflog_logfile="/var/log/pflog"  
28. pflog_flags=""  
29.   
30. pfsync_enable="YES"      # Expose pf state to other hosts for syncing  
31. pfsync_syncdev="vr0"     # Interface for pfsync to work through  
32. pfsync_syncpeer=""       # IP address of pfsync peer host  
33.   
34. relayd_enable="YES"  
35. relayd_flags=""  

三、配置sysctl.conf(加入的相关项)
# vim /etc/sysctl.conf

代码

1. net.inet.carp.preempt=1  
2.   
3. net.inet.tcp.blackhole=2  
4. net.inet.udp.blackhole=1  
5. net.inet.tcp.sendspace=65536  
6. net.inet.tcp.recvspace=65536  

四、配置pf.conf

# vim /etc/pf.conf

代码

1. ################################################################################  
2. # Macro and lists  
3. ################################################################################  
4. lop_if = "lo0"  
5. ext_if = "em0"  
6. int_if = "em1"  
7. sync_if= "vr0"  
8. ext_carp = "carp0"  
9.   
10. icmp_types = "echoreq"  
11.   
12. web_ports = "{ 80, 443 }"  
13.   
14. table persist { 222.222.222.0/24, 221.222.212.0/24 }  
15.   
16. table persist \  
17. { \  
18. 192.168.10.20, \  
19. 192.168.10.21, \  
20. 192.168.10.22, \  
21. 192.168.10.23, \  
22. 192.168.10.24, \  
23. 192.168.10.30, \  
24. 192.168.10.31, \  
25. 192.168.10.32, \  
26. 192.168.10.33, \  
27. 192.168.10.34 \  
28. }  
29.   
30. ################################################################################  
31. # Options, scrub  
32. ################################################################################  
33. set block-policy drop  
34. set limit { states 10000, src-nodes 100000, frags 10000 }  
35. set skip on $lop_if  
36.   
37. scrub in on $ext_if  
38.   
39. ################################################################################  
40. # NAT  
41. ################################################################################  
42. nat on $ext_if from $int_if:network to any -> $ext_if  
43.   
44. ################################################################################  
45. # Redirection  
46. ################################################################################  
47. #rdr on $ext_if proto tcp from any to $ext_carp port $web_ports -> round-robin sticky-address  
48. rdr-anchor "relayd/*"  
49.   
50.   
51. ################################################################################  
52. # Filtering Rules  
53. ################################################################################  
54. block in all  
55. block out all  
56.   
57. pass quick on $lop_if all  
58. #pass quick on $int_if all  
59.   
60. pass quick on { $sync_if } proto pfsync keep state (no-sync)  
61. pass on { $ext_if, $int_if } proto carp keep state  
62.   
63. pass in quick inet proto icmp all icmp-type $icmp_types keep state  
64.   
65. # 这里做了对服务器池的访问策略，如果需要全网开放就将改成any即可  
66. pass in quick on { $ext_if } proto tcp from to $ext_carp port $web_ports flags S/SA modulate state  
67. pass out quick on { $int_if } proto tcp from to port $web_ports flags S/SA modulate state  
68. #还有这里  
69. pass in quick on $ext_if inet from to any keep state  
70.   
71. pass in quick on $int_if inet from any to $int_if:network keep state  
72. pass out quick on $int_if inet from $int_if:network to any keep state  
73.   
74. pass out quick on $ext_if proto tcp all flags S/SA modulate state  
75. pass out quick on $ext_if proto { udp, icmp } all keep state  

五、配置relayd.conf

# vim /usr/etc/relayd.conf

PHP代码

1. ##################  
2. # MACROS          #  
3. ##################  
4. #www1="192.168.10.20, 192.168.10.21"  
5. #www2="192.168.10.30, 192.168.10.31"  
6.   
7. ##### Global Options #####  
8. interval 10  
9. prefork 10  
10.   
11. ##################  
12. # TABLES          #  
13. ##################  
14. #这里就是你的服务器池的IP了。  
15. table { \  
16. 192.168.10.20, \  
17. 192.168.10.21, \  
18. 192.168.10.22, \  
19. 192.168.10.23, \  
20. 192.168.10.24, \  
21. 192.168.10.30, \  
22. 192.168.10.31, \  
23. 192.168.10.32, \  
24. 192.168.10.33, \  
25. 192.168.10.34 \  
26. }  
27.   
28. table { 127.0.0.1 }  
29.   
30. ##### REDIRECTIONS #####  
31. redirect "www" {  
32.      #注意这里的网口是ext_if。  
33.      listen on 192.168.1.51 port 80 interface em0  
34.      #forward to mode roundrobin check http "/" code 200  
35.      #forward to port 80 mode roundrobin sticky-address check tcp  
36.      #因为服务器池的应用是动态的，在重定向时我用了sticky-address来保证同个源请求到原目标服务器。并利用tcp对服务器池进行健康检查。  
37.      tag REDIRECTED  
38.      sticky-address  
39.      forward to port 80 mode roundrobin timeout 300 check tcp  
40.      forward to check icmp  
41. }  

六、启动服务
# /usr/local/etc/rc.d/relayd start
# pfctl -Fa -f /etc/pf.conf

测试, 通过relayctl可以看到服务器池的健康情况，不健康的就不会进行调度了。relayctl命令还有一些比较实用的用法，请自行man 了。

# relayctl show summary

PHP代码

1. Id       Type             Name                             Avlblty Status  
2. 0        redirect         www                                      active  
3. 2        table            www_pool:80                              active (5 hosts up)  
4. 20       host             192.168.10.20                    99.95%   up  
5. 19       host             192.168.10.21                    99.95%   up  
6. 18       host             192.168.10.22                    99.94%   up  
7. 17       host             192.168.10.23                    99.95%   up  
8. 16       host             192.168.10.24                    99.94%   up  
9. 15       host             192.168.10.30                    74.62%   down  
10. 14       host             192.168.10.31                    74.62%   down  
11. 13       host             192.168.10.32                    74.62%   down  
12. 12       host             192.168.10.33                    74.61%   down  
13. 11       host             192.168.10.34                    74.61%   down  
14. 3        table            fallback:80                              active (1 hosts up)  
15. 21       host             127.0.0.1                        100.00% up  

七、服务器池的配置在<>上已讲过，配置是一样的。

八、LoadBalance 的master和slave的配置以<>上的方法来配置。

Tags: relayd, pf, carp, pfsync, loadbalance

出处 http://blog.sysv.cn/?action=showid=14