容器云负载均衡之三：使用iptables对IPVS Direct Routing模式进行端口转换
一、前言
在cloud环境中，不是所有的application都有使用80端口这种特权端口的权限；但是在IPVS direct routing模式下，VIP监听的端口必须要保持和后端real server上的application监听的端口一致，见文章http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.rewrite_ports.html：
本文通过iptables来解决这个问题，最终达到IPVS DR模式监听80端口，后端HTTP服务监听8080端口，客户端访问VIP:80可以得到后端8080端口提供的服务。
转载自https://blog.csdn.net/cloudvtech
二、IPVS DR模式端口转换的实现
2.1 机器配置
IPVS director： 192.168.166.102
IPVS real server：192.168.166.103
VIP：192.168.166.111 2.2 IPVS director的设置
ipvsadm -C
ipvsadm -A -t 192.168.166.111 -s rr
ipvsadm -a -t 192.168.166.111:80 -r 192.168.166.103:80 -w 1 -g
ifconfig ens33:0 192.168.166.111 broadcast 192.168.166.255 netmask 255.255.255.0 up
route add -host 192.168.166.111 dev ens33:0 /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
net.ipv4.conf.all.rp_filter = 02.3 IPVS real server的设置
/etc/sysctl.conf
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.ens33.arp_ignore = 1
net.ipv4.conf.ens33.arp_announce = 2
ifconfig lo:0 192.168.166.111 broadcast 192.168.166.255 netmask 255.255.255.255 up
route add -host 192.168.166.111 dev lo:0 并且启动http服务监听在80端口
2.4 在IPVS director插入如下iptables debug LOG
iptables -t mangle -A INPUT -p tcp -m tcp --dport 80 -j LOG --log-prefix "[INPUT|mangle] "
iptables -t mangle -A OUTPUT -p tcp -m tcp --dport 80 -j LOG --log-prefix "[OUTPUT|mangle] "2.5 在IPVS read server插入如下iptables debug LOG
iptables -t mangle -A OUTPUT -p tcp -m tcp --sport 8080 -j LOG --log-prefix "[OUTPUT|mangle] port 8080 "
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j LOG --log-prefix "[INPUT|nat] before REDIRECT "
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 80802.6 从其它客户端访问VIP的80端口
curl 192.168.166.111:80
可以得到返回：
curl 192.168.166.111 -vvv
* Rebuilt URL to: 192.168.166.111/
* Trying 192.168.166.111...
* TCP_NODELAY set
* Connected to 192.168.166.111 (192.168.166.111) port 80 (#0)
> GET / HTTP/1.1
> Host: 192.168.166.111
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Date: Tue, 01 May 2018 05:37:41 GMT
< Server: Apache/2.4.6 (CentOS)
< Last-Modified: Thu, 16 Oct 2014 13:20:58 GMT
< ETag: "1321-5058a1e728280"
< Accept-Ranges: bytes
< Content-Length: 4897
< Content-Type: text/html; charset=UTF-8
<
2.7 IPVS director iptables LOG输出
May 1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=51301 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
May 1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=51301 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
May 1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=12788 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4117 RES=0x00 ACK URGP=0
May 1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=12788 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4117 RES=0x00 ACK URGP=0
May 1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=43668 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4117 RES=0x00 ACK PSH URGP=0
May 1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=43668 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4117 RES=0x00 ACK PSH URGP=0
May 1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=61495 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4027 RES=0x00 ACK URGP=0
May 1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=61495 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4027 RES=0x00 ACK URGP=0
May 1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=11136 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK URGP=0
May 1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=11136 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK URGP=0
May 1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=10635 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4070 RES=0x00 ACK URGP=0
May 1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=10635 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4070 RES=0x00 ACK URGP=0
May 1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=46543 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK FIN URGP=0
May 1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=46543 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK FIN URGP=0
May 1 01:38:04 k8s-node1 kernel: [INPUT|mangle] IN=ens33 OUT= MAC=00:0c:29:e7:54:7c:00:50:56:c0:00:08:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=53957 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK URGP=0
May 1 01:38:04 k8s-node1 kernel: [OUTPUT|mangle] IN= OUT=ens33 SRC=192.168.166.1 DST=192.168.166.111 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=53957 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=4096 RES=0x00 ACK URGP=0
2.8 IPVS real server iptables LOG输出
May 1 01:37:41 k8s-node2 kernel: [INPUT|nat] before REDIRECT IN=ens33 OUT= MAC=00:0c:29:75:24:37:00:0c:29:e7:54:7c:08:00 SRC=192.168.166.1 DST=192.168.166.111 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=51301 DF PROTO=TCP SPT=56047 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
May 1 01:37:41 k8s-node2 kernel: [OUTPUT|mangle] port 8080 IN= OUT=ens33 SRC=192.168.166.103 DST=192.168.166.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=8080 DPT=56047 WINDOW=28960 RES=0x00 ACK SYN URGP=0
May 1 01:37:41 k8s-node2 kernel: [OUTPUT|mangle] port 8080 IN= OUT=ens33 SRC=192.168.166.103 DST=192.168.166.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=32293 DF PROTO=TCP SPT=8080 DPT=56047 WINDOW=227 RES=0x00 ACK URGP=0
May 1 01:37:41 k8s-node2 kernel: [OUTPUT|mangle] port 8080 IN= OUT=ens33 SRC=192.168.166.103 DST=192.168.166.1 LEN=4396 TOS=0x00 PREC=0x00 TTL=64 ID=32294 DF PROTO=TCP SPT=8080 DPT=56047 WINDOW=227 RES=0x00 ACK URGP=0
May 1 01:37:41 k8s-node2 kernel: [OUTPUT|mangle] port 8080 IN= OUT=ens33 SRC=192.168.166.103 DST=192.168.166.1 LEN=857 TOS=0x00 PREC=0x00 TTL=64 ID=32297 DF PROTO=TCP SPT=8080 DPT=56047 WINDOW=227 RES=0x00 ACK PSH URGP=0
May 1 01:37:41 k8s-node2 kernel: [OUTPUT|mangle] port 8080 IN= OUT=ens33 SRC=192.168.166.103 DST=192.168.166.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=32298 DF PROTO=TCP SPT=8080 DPT=56047 WINDOW=227 RES=0x00 ACK FIN URGP=0 转载自https://blog.csdn.net/cloudvtech
三、与从IPVS director访问VIP相结合
可以将本方案和根据文章《从IPVS DR模式下director不能访问VIP问题的探究》提供的解决方案绑定在一起，就可以为kubernetes容器云环境下的IPVS service找到无缝的解决方案：
- IPVS director可以和后端application混合部署
- IPVS director可以监听80特权端口
- application只需监听任意无特权端口
- 客户端可以在包括IPVS director所在的worker node之内的任何位置访问VIP:80服务
转载自https://blog.csdn.net/cloudvtech