kubernetes系列之一：Kubernetes如何利用iptables对外暴露service
一、前言
Kubernetes利用iptables达成以下两个目的：
- 对外暴露POD和服务(通过host port、cluster service、node external IP、external load balancer、node port)
- 简单的负载均衡
在Kubernetes worker node的iptables的NAT表的prerouting和output 链表会出现如下规则作为kubernetes相关的数据包操作的入口：
- -A PREROUTING -m comment --comment "kube hostport portals" -m addrtype --dst-type LOCAL -j KUBE-HOSTPORTS
- -A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
- -A PREROUTING -m comment --comment "handle ClusterIPs; NOTE: this must be before the NodePort rules" -j KUBE-PORTALS-CONTAINER
- -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
- -A PREROUTING -m addrtype --dst-type LOCAL -m comment --comment "handle service NodePorts; NOTE: this must be the last rule in the chain" -j KUBE-NODEPORT-CONTAINER
- -A OUTPUT -m comment --comment "kube hostport portals" -m addrtype --dst-type LOCAL -j KUBE-HOSTPORTS
- -A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
- -A OUTPUT -m comment --comment "handle ClusterIPs; NOTE: this must be before the NodePort rules" -j KUBE-PORTALS-HOST
- -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
- -A OUTPUT -m addrtype --dst-type LOCAL -m comment --comment "handle service NodePorts; NOTE: this must be the last rule in the chain" -j KUBE-NODEPORT-HOST
PREROUTING链表用来处理外部进来的数据包(包括来自外部网络的和来自本机container的)
规则1 用来向使用hostport的POD转发数据包 (用于IPTables模式)
规则2 用来向kubernetes服务(cluster service/nodeport service/load balancer service)转发数据包 (用于IPTables模式)
规则3 用来处理容器内向cluster service虚IP发出的请求 (用于kube-proxy模式)
规则5 用来处理容器向nodeport发出的请求 (用于kube-proxy模式)
OUTPUT链表用来处理发向外部的数据包(包括发到外部网络的和发到本机上container的)
规则6 用来处理主机向host port发出的请求 (用于IPTables模式)
规则7 用来处理主机向kubernetes service发出的请求 (用于IPTables模式)
规则8 用来处理主机向cluster service虚IP发出的请求 (用于kube-proxy模式)
规则10 用来处理主机向nodeport发出的请求 (用于kube-proxy模式)
kubernetes各种类型的服务对外暴露的顺序依次是hostport、cluster service、node external、loadbalancer service和nodeport service。
-A KUBE-SERVICES -d cluster_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name cluster IP" -m tcp --dport 80 -j KUBE-SVC-SSSSSS
-A KUBE-SERVICES -d node_external_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name external IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d node_external_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name external IP" -m tcp --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type LOCAL -j KUBE-SVC-SSSSSS
-A KUBE-SERVICES -d node_external_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name external IP" -m tcp --dport 80 -m addrtype --dst-type LOCAL -j KUBE-SVC-SSSSSS
-A KUBE-SERVICES -d loadbalancer_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-SSSSSS
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS转载自https://blog.csdn.net/cloudvtech
二、cluster service如何使用iptables对外暴露POD服务
1. cluster service的入口是这个规则"-A [PREROUTING|OUTPUT] -m comment --comment "kubernetes service portals" -j KUBE-SERVICES"，跳到KUBE-SERVICE chain
2. 这个KUBE-SERVICE chain由一些列满足如下规则的rule组成：
满足访问某个cluster_VIP和port的请求将会被倒入到rule KUBE-SVC-XXXXXX
-A KUBE-SERVICES -d cluster_VIP/32 -p tcp -m comment --comment "namespace/pod_name:port_name cluster IP" -m tcp --dport service_port -j KUBE-SVC-XXXXXX
3. KUBE-SVC-XXXXXX的组成如下：
-A KUBE-SVC-XXXXXX -m comment --comment "namespace/pod_name:port_name" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-AAAAAA
-A KUBE-SVC-XXXXXX -m comment --comment "namespace/pod_name:port_name" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-BBBBBB
-A KUBE-SVC-XXXXXX -m comment --comment "namespace/pod_name:port_name" -j KUBE-SEP-CCCCCC会按照概率对所有后端的POD进行选择转发
4. KUBE-SEP-CCCCC的组成如下：
-A KUBE-SEP-CCCCCC -s POD_IP/32 -m comment --comment "namespace/pod_name:port_name" -j KUBE-MARK-MASQ
-A KUBE-SEP-CCCCCC -p tcp -m comment --comment "namespace/pod_name:port_name" -m tcp -j DNAT --to-destination POD_IP:PORT需要进行一次DNAT，把数据包定位到选定POD，然后经由路由进入远端或者本地的POD
转载自https://blog.csdn.net/cloudvtech
三、external load balancer serice如何使用iptables对外暴露POD服务
1. 经由external load balancer转发的外部请求会带有外部IP地址，这个地址会匹配如下规则：-A KUBE-SERVICES -d loadbalancer_IP/32 -p tcp -m comment --comment "namespace/pod_name:port_name loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-SSSSSS2. KUBE-FW-SSSSSS组成如下：
-A KUBE-FW-SSSSSS -m comment --comment "namespace/pod_name:port_name loadbalancer IP" -j KUBE-XLB-KKKKKK
-A KUBE-FW-SSSSSS -m comment --comment "namespace/pod_name:port_name loadbalancer IP" -j KUBE-MARK-DROP会jump到为external访问生成的load balance规则KUBE-XLB-KKKKKK
3.KUBE-XLB-KKKKKK由一系列概率选择规则组成：
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 0 for namespace/pod_name:port_name" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-AAAAAA
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 1 for namespace/pod_name:port_name" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-BBBBBB
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 2 for namespace/pod_name:port_name" -j KUBE-SEP-CCCCCC将数据包导到具体的POD相关iptables规则
4. KUBE-SEP-CCCCC的组成如下：
-A KUBE-SEP-CCCCCC -s POD_IP/32 -m comment --comment "namespace/pod_name:port_name" -j KUBE-MARK-MASQ
-A KUBE-SEP-CCCCCC -p tcp -m comment --comment "namespace/pod_name:port_name" -m tcp -j DNAT --to-destination POD_IP:PORT需要进行一次DNAT，把数据包定位到选定POD，然后经由路由进入远端或者本地的POD
转载自https://blog.csdn.net/cloudvtech
四、nodePort serice如何使用iptables对外暴露POD服务
1. KUBE-SERVICE chain的最末端将跳转到nodeport service对应的chain KUBE-NODEPORTS
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-NODEPORTS -p tcp -m comment --comment "namespace/pod_name:port_name" -m tcp --dport 80 -j KUBE-XLB-KKKKKK-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 0 for namespace/pod_name:port_name" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-AAAAAA
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 1 for namespace/pod_name:port_name" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-BBBBBB
-A KUBE-XLB-KKKKKK -m comment --comment "Balancing rule 2 for namespace/pod_name:port_name" -j KUBE-SEP-CCCCCC将数据包导到具体的POD相关iptables规则
4. KUBE-SEP-CCCCC的组成如下：
-A KUBE-SEP-CCCCCC -s POD_IP/32 -m comment --comment "namespace/pod_name:port_name" -j KUBE-MARK-MASQ
-A KUBE-SEP-CCCCCC -p tcp -m comment --comment "namespace/pod_name:port_name" -m tcp -j DNAT --to-destination POD_IP:PORT需要进行一次DNAT，把数据包定位到选定POD，然后经由路由进入远端或者本地的POD
转载自https://blog.csdn.net/cloudvtech