Openshift的网络之六：Router和ipfailover

一、前言

在文章《Kubernetes如何利用iptables对外暴露service》中提到了部署在Kubernetes上的服务对外暴露的方法，比如通过host port、cluster service、node external IP、external load balancer、node port等，这都是借助于iptables的功能在三层和四层进行的。

在Openshift中，提供了另外一种基于七层的后端服务暴露机制，那就是Openshift Router。Router是一个基于HAProxy的应用层代理，启动之后监听在POD宿主机机的端口，当请求过来之后，会由HAProxy根据URL或者内容进行后端POD的选则，发送代理请求，再将响应发送到客户端。

为了保证后端服务访问接口的健壮性，需要启动多个router的replica，然后将这些replica组成一个cluster，共享一个VIP，提供active-standby的服务模式，这个模式需要启动Openshift的ipfailover功能。

ipfailover管理一个VIP的池，每个VIP都代表着后端一系列POD或者一个service，VIP绑定在任何一个或者多个健康的工作节点上。ipfailover借助keepalived，使用VRRP协议通过IP floating保证VIP永远绑定在健康的节点上，只要后端服务是健康的，VIP就可以提供对外的服务。

载自https://blog.csdn.net/cloudvtech

二、Openshift Router和IPFailover一起工作

多个Openshift Router可以组成一个cluster，这个cluster可以为多个VIP提供接入服务。多个Router分布在多个工作节点上，VIP可以绑定在有router运行的任意一个工作节点上，一个工作节点只有一个router但是可以绑定多个VIP。

这个方案的处理流程如下：

• keepalived在所有运行Router的节点之间相互通信，为每个VIP选择一个active master对外advertise这个VIP和MAC
• 客户端向VIP发送的请求到达之后，由HAProxy进行处理
• HAProxy根据URL和请求内容，选择合适的后端POD
• 通过iptables进行四层数据包NAT然后由OVS SDN将数据包转送到POD所在节点
• 在POD所在节点经过NAT操作进入POD
• POD处理请求并产生返回数据包
• 数据包经上述步骤的反方向操作返回接入HAProxy
• HAProxy将数据返回给客户端

载自https://blog.csdn.net/cloudvtech

三、在Openshift配置Router和IPFailover

3.1 Openshift运行环境

Master: 200.222.0.72
node1: 200.222.0.73
node2: 200.222.0.74
docker registry: 200.222.243

3.2 为node打label

oc label nodes node1.os.com "ipf=yes"
oc label nodes node2.os.com "ipf=yes”

oc get nodes --selector="ipf=yes"
NAME           STATUS    AGE
node1.os.com   Ready     7d
node2.os.com   Ready     7d 

3.3 建立一个Openshift project并建立一个node port service

oc login -u system:admin -n ipf-test
kubectl run source-ip-app --image=gcr.io/google_containers/echoserver:1.4
kubectl expose deployment source-ip-app --name=nodeport --port=8080 --target-port=8080 --type=NodePort 

在Openshift GUI设置这个deployment的replica为2

3.4 在Openshift GUI设置Router的数目为2

3.5 为node port service设置route

这个操作将使得HAProxy可以根据后端的POD的变化修改的配置

载自https://blog.csdn.net/cloudvtech

3.6 启动Openshift IPFailover

oadm policy add-scc-to-user privileged -z router
oadm ipfailover ipf-ha-router-primary     --replicas=2     --watch-port=8080     --selector="ipf=yes"     --virtual-ips="200.222.0.185-186"     --service-account=router     --create 

3.7 访问VIP

修改/etc/hosts加入：

200.222.0.185  nodeport-ipf-test.router.default.svc.cluster.local

或者200.222.0.186  nodeport-ipf-test.router.default.svc.cluster.local

curl nodeport-ipf-test.router.default.svc.cluster.local/ipf

可以看到请求会轮流进入两个POD

3.8 一些日志

root@node2 ~]# docker logs -f 942a9a0f0ecc
  - Loading ip_vs module ...
  - Checking if ip_vs module is available ...
ip_vs                 141092  0
  - Module ip_vs is loaded.
  - check for iptables rule for keepalived multicast (224.0.0.18) ...
  - Generating and writing config to /etc/keepalived/keepalived.conf
  - Starting failover services ...
Starting Healthcheck child process, pid=112
Initializing ipvs 2.6
Starting VRRP child process, pid=113
Netlink reflector reports IP 10.79.47.144 added
Netlink reflector reports IP 200.222.0.74 added
Netlink reflector reports IP 200.222.0.185 added
Netlink reflector reports IP 172.17.0.1 added
Netlink reflector reports IP 10.128.0.1 added
Netlink reflector reports IP 10.79.47.144 added
Netlink reflector reports IP fe80::6ee6:ea71:3f4d:b641 added
Netlink reflector reports IP fe80::828e:c0f4:d54e:de2 added
Netlink reflector reports IP fe80::16e6:7310:9226:467e added
Netlink reflector reports IP fe80::3c44:b7fd:2c7f:9189 added
Netlink reflector reports IP 200.222.0.74 added
Netlink reflector reports IP fe80::5840:b3ff:fe3c:d731 added
Netlink reflector reports IP fe80::24d2:3eff:fe34:dcec added
Netlink reflector reports IP 200.222.0.185 added
Netlink reflector reports IP 172.17.0.1 added
Registering Kernel netlink reflector
Netlink reflector reports IP 10.128.0.1 added
Registering Kernel netlink command channel
Netlink reflector reports IP fe80::6ee6:ea71:3f4d:b641 added
Netlink reflector reports IP fe80::828e:c0f4:d54e:de2 added
Opening file '/etc/keepalived/keepalived.conf'.
Netlink reflector reports IP fe80::16e6:7310:9226:467e added
Netlink reflector reports IP fe80::3c44:b7fd:2c7f:9189 added
Netlink reflector reports IP fe80::5840:b3ff:fe3c:d731 added
Netlink reflector reports IP fe80::24d2:3eff:fe34:dcec added
Registering Kernel netlink reflector
Registering Kernel netlink command channel
Configuration is using : 8923 Bytes
Registering gratuitous ARP shared channel
Opening file '/etc/keepalived/keepalived.conf'.
Truncating auth_pass to 8 characters
Truncating auth_pass to 8 characters
Configuration is using : 71477 Bytes
Using LinkWatch kernel netlink reflector...
VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(9,10)]
Using LinkWatch kernel netlink reflector...
VRRP_Script(chk_ipf_ha_router_primary) succeeded
VRRP_Instance(ipf_ha_router_primary_VIP_1) Transition to MASTER STATE
VRRP_Instance(ipf_ha_router_primary_VIP_1) Received lower prio advert, forcing new election
VRRP_Instance(ipf_ha_router_primary_VIP_2) Transition to MASTER STATE
VRRP_Instance(ipf_ha_router_primary_VIP_2) Received higher prio advert
VRRP_Instance(ipf_ha_router_primary_VIP_2) Entering BACKUP STATE
VRRP_Instance(ipf_ha_router_primary_VIP_1) Entering MASTER STATE
VRRP_Instance(ipf_ha_router_primary_VIP_1) setting protocol VIPs.
VRRP_Instance(ipf_ha_router_primary_VIP_1) Sending gratuitous ARPs on ens160 for 200.222.0.185
Netlink reflector reports IP 200.222.0.185 added
VRRP_Instance(ipf_ha_router_primary_VIP_1) Sending gratuitous ARPs on ens160 for 200.222.0.185 

载自https://blog.csdn.net/cloudvtech