K8S 网络策略

1. 网络策略

NetworkPolicy 是一种以应用为中心的结构，允许你设置如何允许 Pod 与网络上的各类网络 “实体” 通信，在 IP/Port (L3/L4) 层面控制网络流量，用于隔离应用以减少攻击面。

Pod 之间能否通信，可通过如下三个组合进行确认：

• 其他被允许的 Pods (例如：Pod 无法限制对自身的访问)
• 被允许的 namespace
• IP CIDR (例如：与Pod运行所在节点的节点的通信总是被允许的)

支持的网络插件：

• Calico
• Canal
• Cilium
• Kube-router
• Romana
• Weave Net

网络策略定义：

• Pod Selector：A group of pods is selected through labels (such as application name). The selected group of pods is isolated and explicit rules for allowed source, and destination communication are applied to the group.

• Ingress:

  • NetworkPolicyPeer: is the other side of the connection
    • If an IP Block is specified, the rule applies to the addresses defined by the block.
    • If a Namespace is specified, the rule applies to select pods within the namespace.
    • If a Namespace Selector is specified, the rule applies to all pods within a namespace.
    • If a Pod and Namespace Selector is specified, the rule applies to select pods within a specified namespace.
  • NetworkPolicyPort：allows you to explicitly name ingress and egress ports or protocols that may communicate with the pod group.

• Engress:

  • NetworkPolicyPeer
  • NetworkPolicyPort

2. NetworkPolicy 资源

默认情况下，Pod是非隔离的，可接收任何流量

Pod 在被某 NetworkPolicy 选中时进入隔离状态。一旦 namespace 中有 NetworkPolicy 选择了特定的 Pod，该 Pod 会拒绝该 NetworkPolicy 所不允许的连接。

网络策略不会冲突：任何一个或多个策略选择了一个 Pod，该 Pod 受限于这些策略的 Ingress/Egress 规则的并集

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: network-policy-sample
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    - namespaceSelector:
        matchLabels:
          project: myproject
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

podSelector：通过 label 方式选择适用该策略的 Pod，如果为空，则选择该 namespace 下的所有 Pod

policyTypes：流量类型，Ingree或Egress，默认Ingress

ingress：入口规则白名单列表。每条规则允许同时匹配 from 和 ports 部分的流量。示例：它匹配某个特定的端口，第一个通过 ipBlock 指定，第二个通过 namespaceSelector 指定，第三个通过 podSelector 指定

egress：出口规则白名单列表。每条规则允许都匹配 to 和 port 两部分流量。示例：它指定端口的流量匹配到 10.0.0.0/24中的任何目的地址

示例网络策略总结：

• 隔离 “default” 名字空间下 “role=db” 的 Pod （如果它们不是已经被隔离的话）。

• （Ingress 规则）允许以下 Pod 连接到 “default” 名字空间下的带有 “role=db” 标签的所有 Pod 的 6379 TCP 端口：

  • “default” 名字空间下带有 “role=frontend” 标签的所有 Pod

  • 带有 “project=myproject” 标签的所有名字空间中的 Pod

  • IP 地址范围为 172.17.0.0–172.17.0.255 和 172.17.2.0–172.17.255.255 （即，除了 172.17.1.0/24 之外的所有 172.17.0.0/16）

• （Egress 规则）允许从带有 “role=db” 标签的名字空间下的任何 Pod 到 CIDR 10.0.0.0/24 下 5978 TCP 端口的连接。

3. 选择器 to 和 from 的行为

podSelector：在与 NetworkPolicy 相同的名字空间中选择特定的 Pod，应将其允许作为入站流量来源或出站流量目的地。

namespaceSelector：选择特定的名字空间，应将所有 Pod 用作其 入站流量来源或出站流量目的地。

ipBlock: 选择特定的 IP CIDR 范围以用作入站流量来源或出站流量目的地。 这些应该是集群外部 IP，因为 Pod IP 存在时间短暂的且随机产生。

  ...
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          user: alice
    - podSelector:
        matchLabels:
          role: client
  ...

4. 默认策略

4.1 拒绝所有入站流量

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress: []  # 可省略

4.2 允许所有入站流量

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress
spec:
  podSelector: {}
  ingress:
  - {}
  policyTypes:
  - Ingress

4.3 拒绝所有出站流量

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress

4.4 允许所有出站流量

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-egress
spec:
  podSelector: {}
  egress:
  - {}
  policyTypes:
  - Egress

4.5 拒绝所有入口和所有出站流量

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

5. 应用场景

5.1 Deny all traffic to an application

This NetworkPolicy will drop all traffic to pods of an application, selected using Pod Selectors.

Use Cases:

• It’s very common: To start whitelisting the traffic using Network Policies, first you need to blacklist the traffic using this policy.
• You want to run a Pod and want to prevent any other Pods communicating with it.
• You temporarily want to isolate traffic to a Service from other Pods.

$ kubectl run web --image=nginx --labels="app=web" --expose --port=80

$ kubectl run -it test-$RANDOM --rm --image=alpine -- sh
/ # wget -qO- http://web
<!DOCTYPE html>
<html>
<head>
...

# 增加网络策略，限制访问
$ cat <<EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-deny-all
spec:
  podSelector:
    matchLabels:
      app: web
  ingress: []   # no rules specified, it causes incoming traffic to be dropped
EOF

$ kubectl apply -f web-deny-all.yaml

# 再次尝试访问，timeout
$ kubectl run -it test-$RANDOM --rm --image=alpine -- sh
/ # wget -qO- --timeout=3 http://web
wget: download timed out

# 清理
kubectl delete pod,svc web
kubectl delete networkpolicy web-deny-all

5.2 Limit traffic to an application

You can create Networking Policies allowing traffic from only certain Pods.

Use Case:

• Restrict traffic to a service only to other microservices that need to use it.
• Restrict connections to a database only to the application using it.

$ kubectl run web --image=nginx --labels="app=bookstore,role=api" --expose --port=80

$ cat << EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: api-allow
spec:
  podSelector:
    matchLabels:
      app: bookstore
      role: api
  ingress:
  - from:
      - podSelector:
          matchLabels:
            app: bookstore   # only the pod tagged with bookstore can access it
EOF

# 未指定正确的标签，无法访问
$ kubectl run -it test-$RANDOM --rm --image=alpine -- sh
/ # wget -qO- --timeout=2 http://web
wget: download timed out

# 标签正确，可访问
$ kubectl run -it test-$RANDOM --rm --image=alpine --labels="app=bookstore,role=frontend" -- sh
/ # wget -qO- --timeout=2 http://web
<!DOCTYPE html>
<html>
<head>
...

# 清理
kubectl delete pod,svc web
kubectl delete networkpolicy api-allow

5.3 Allow all traffic to an application

Applying this policy makes any other policies restricting the traffic to the pod void, and allow all traffic to it from its namespace and other namespaces.

Use Case: After applying a deny-all policy which blocks all non-whitelisted traffic to the application, now you have to allow access to an application from all pods in the current namespace.

$ kubectl run web --image=nginx --labels="app=web" --expose --port=80

$ cat <<EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-deny-all
spec:
  podSelector:
    matchLabels:
      app: web
  ingress: []
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-allow-all
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
  - {}  # empty, allow all traffic
EOF

# allow的权限大于deny
$ kubectl run -it test-$RANDOM --rm --image=alpine -- sh
/ # wget -qO- --timeout=2 http://web
<!DOCTYPE html>
<html>
<head>
...

# 清理
kubectl delete pod,svc web
kubectl delete networkpolicy web-deny-all web-allow-all

补充：Empty ingress rule ({}) allows traffic from all pods in the current namespace, as well as other namespaces. It corresponds to:

- from:
  - podSelector: {}
    namespaceSelector: {}

5.4 DENY all non-whitelisted traffic to a namespace

Use Case: This is a fundamental policy, blocking all cross-pod networking other than the ones whitelisted via the other Network Policies you deploy.

Consider applying this manifest to any namespace you deploy workloads to (anything but kube-system).

Best Practice: This policy will give you a default “deny all” functionality. This way, you can clearly identify which components have dependency on which components and deploy Network Policies which can be translated to dependency graphs between components.

$ kubectl run web --image=nginx --expose --port=80

$ cat <<EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-deny-all
  namespace: default
spec:
  podSelector: {}   # empty, it will match all the pods
  ingress: []       # no rules specified, it causes incoming traffic to be dropped
EOF

# same namespace
$ kubectl run test-$RANDOM -it --rm --image=alpine -- sh
/ # wget -qO- --timeout=2 http://web
wget: download timed out

# across namespace
$ kubectl run test-$RANDOM -it --rm --image=alpine -n kube-system -- sh
/ # wget -qO- --timeout=2 http://web
wget: bad address 'web'
/ # wget -qO- --timeout=2 http://web.default
wget: download timed out

# 清理
kubectl delete pod,svc web
kubectl delete networkpolicy default-deny-all

5.5 DENY all traffic from other namespaces

You can configure a NetworkPolicy to deny all the traffic from other namespaces while allowing all the traffic coming from the same namespace the pod deployed to.

Use Cases

• You do not want deployments in test namespace to accidentally send traffic to other services or databases in prod namespace.
• You host applications from different customers in separate Kubernetes namespaces and you would like to block traffic coming from outside a namespace.

$ kubectl run web --image=nginx --labels="app=web" --namespace=default --expose --port=80

$ cat <<EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  namespace: default
  name: deny-from-other-namespaces
spec:
  podSelector:
    matchLabels:   # empty, select all pods in 'default' namespace
  ingress:
  - from:
    - podSelector: {}   # empty, select all pods in 'default' namespace
EOF

# 非default命名空间，无法访问
$ kubectl create namespace foo
$ kubectl run test-$RANDOM -it --rm --image=alpine -n foo -- sh
/ # wget -qO- --timeout=2 http://web.default
wget: download timed out

# default命名空间，可正常访问
$ kubectl run test-$RANDOM -it --rm --image=alpine -n default -- sh
/ # wget -qO- --timeout=2 http://web
<!DOCTYPE html>
<html>
<head>
...

# 清理
kubectl delete pod,svc web
kubectl delete ns foo
kubectl delete networkpolicy deny-from-other-namespaces

5.5 ALLOW traffic to an application from all namespaces

This NetworkPolicy will allow traffic from all pods in all namespaces to a particular application.

Use Case:

• You have a common service or a database which is used by deployments in different namespaces.

$ kubectl run web --image=nginx --labels="app=web" --namespace=default --expose --port 80

$ cat <<EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  namespace: default
  name: web-allow-all-namespaces
spec:
  podSelector:
    matchLabels:
      app: web    # apply the policy only to the pods that are labeled by app=web
  ingress:
  - from:
    - namespaceSelector: {}  # empty, allow all namespaces to access
EOF

$ kubectl run test-$RANDOM -it --rm --image=alpine --namespace=bar -- sh
/ # wget -qO- --timeout=2 http://web.default
<!DOCTYPE html>
<html>
<head>
...

# 清理
kubectl delete pod,svc web
kubectl delete ns bar
kubectl delete networkpolicy web-allow-all-namespaces

5.6 ALLOW all traffic from a namespace

This policy is similar to allowing traffic from all namespaces but shows how you can choose particular namespaces.

Use Case:

• Restrict traffic to a production database only to namespaces where production workloads are deployed.
• Enable monitoring tools deployed to a particular namespace to scrape metrics from the current namespace.

$ kubectl run web --image=nginx --labels="app=web" --expose --port 80

$ kubectl create namespace dev
$ kubectl label namespace/dev purpose=develop
$ kubectl create namespace prod
$ kubectl label namespace/prod purpose=production

$ cat <<EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-allow-prod
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          purpose: production   # only pods in namespace that has label `purpose=prodution` will be allowed 
EOF

$ kubectl run test-$RANDOM -it --rm --image=alpine --namespace=dev -- sh
/ # wget -qO- --timeout=2 http://web.default
wget: download timed out

$ kubectl run test-$RANDOM -it --rm --image=alpine --namespace=prod -- sh
/ # wget -qO- --timeout=2 http://web.default
<!DOCTYPE html>
<html>
<head>
...

# 清理
kubectl delete pod,svc web
kubectl delete ns dev prod
kubectl delete networkpolicy web-allow-prod

5.7 ALLOW traffic from some pods in another namespace

$ kubectl run web --image=nginx --labels="app=web" --expose --port=80

$ kubectl create namespace other
$ kubectl label namespace/other team=operations

$ cat <<EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-allow-all-ns-monitoring
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
    - from:
      - namespaceSelector:     
          matchLabels:
            team: operations  # chooses all pods in namespaces labelled with team=operations
        podSelector:          
          matchLabels:
            type: monitoring  # chooses pods with type=monitoring
EOF

# 仅namespace满足，不能访问
$ kubectl run test-$RANDOM -it --rm --image=alpine --namespace=other -- sh
/ # wget -qO- --timeout=2 http://web.default
wget: download timed out

# 仅label满足，不能访问
$ kubectl run test-$RANDOM -it --rm --image=alpine --labels="type=monitoring" -- sh
/ # wget -qO- --timeout=2 http://web.default
wget: download timed out

# namespace和label都满足，可以访问
$ kubectl run test-$RANDOM -it --rm --image=alpine --namespace=other --labels="type=monitoring" -- sh
If you don't see a command prompt, try pressing enter.
/ # wget -qO- --timeout=2 http://web.default
<!DOCTYPE html>
<html>
<head>
...

# 清理
kubectl delete pod,svc web
kubectl delete ns other
kubectl delete networkpolicy web-allow-all-ns-monitoring

5.8 ALLOW traffic from external clients

This Network Policy enables external clients from the public Internet directly or via a Load Balancer to access to the pod.

Use Cases:

• You need to expose the pods to the public Internet in a namespace denying all non-whitelisted traffic

$ kubectl run web --image=nginx --labels="app=web" --port 80
$ kubectl expose pod/web --type=NodePort

$ cat <<EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: web-allow-external
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
  - {}
EOF

$ kubectl get svc web
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
web          NodePort    10.96.223.48   <none>        80:31261/TCP   65s

$ wget -qO- http://192.168.3.103:31261
<!DOCTYPE html>
<html>
<head>
...

# 清理
kubectl delete pod,svc web
kubectl delete networkpolicy web-allow-external

5.9 ALLOW traffic only to a port of an application

This NetworkPolicy lets you define ingress rules for specific ports of an application. If you do not specify a port in the ingress rules, the rule applies to all ports.

A port may be either a numerical or named port on a pod.

Use Cases

• Allow monitoring system to collect the metrics by querying the diagnostics port of your application, without giving it access to the rest of the application.

$ kubectl run web --image=ahmet/app-on-two-ports --labels="app=web"
$ kubectl create service clusterip web --tcp 8001:8000 -tcp 5001:5000

$ cat <<EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: api-allow-5000
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
  - ports:
    - port: 5000
    from:
    - podSelector:
        matchLabels:
          role: monitoring
EOF

# 未指定label，无法访问
$ kubectl run test-$RANDOM -it --rm --image=alpine -- sh
/ # wget -qO- --timeout=2 http://web:5001/metrics
wget: download timed out

# 指定label，且只能访问5000端口
$ kubectl run test-$RANDOM -it --rm --image=alpine --labels="role=monitoring" -- sh
/ # wget -qO- --timeout=2 http://web:8001
wget: download timed out
/ # wget -qO- --timeout=2 http://web:5001/metrics
http.requests=1
go.goroutines=5
go.cpus=4

# 清理
kubectl delete pod,svc web
kubectl delete networkpolicy api-allow-5000

5.10 ALLOW traffic from apps using multiple selectors

NetworkPolicy lets you define multiple pod selectors to allow traffic from.

Use Case

• Create a combined NetworkPolicy that has the list of microservices that are allowed to connect to an application.

$ kubectl run db --image=redis --labels="app=bookstore,role=db" --expose --port=6379

$ cat <<EOF | kubectl apply -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: redis-allow-services
spec:
  podSelector:
    matchLabels:
      app: bookstore
      role: db
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: bookstore
          role: search
    - podSelector:
        matchLabels:
          app: bookstore
          role: api
    - podSelector:
        matchLabels:
          app: inventory
          role: web
EOF

# label 满足条件
$ kubectl run test-$RANDOM -it --rm --image=alpine --labels="app=inventory,role=web" -- sh
/ # nc -v -w 2 db 6379
db (10.96.204.248:6379) open

# label 不满足
$ kubectl run test-$RANDOM -it --rm --image=alpine --labels="app=other" -- sh
/ # nc -v -w 2 db 6379
nc: db (10.96.204.248:6379): Operation timed out

# 清理资源
kubectl delete pod,svc db
kubectl delete networkpolicy redis-allow-services

5.11 DENY egress traffic from an application

Use Cases:

• You want to prevent an application from establishing any connections to outside of the Pod.
• Useful for restricting outbound traffic of single-instance databases and datastores.

$ kubectl run web --image=nginx --labels="app=web" --expose --port=80

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foo-deny-egress
spec:
  podSelector:
    matchLabels:
      app: foo
  policyTypes:
  - Egress
  egress: []
EOF

$ kubectl run test-$RANDOM -it --rm --image=alpine --labels="app=foo" -- sh
/ # wget -qO- --timeout=2 http://web
wget: bad address 'web'
/ # wget -qO- --timeout=2 http://google.com
wget: bad address 'google.com'
/ # ping -c 3 -W 2 192.168.3.1
PING 192.168.3.1 (192.168.3.1): 56 data bytes
--- 192.168.3.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
/ # ping -W 2 -c 3 google.com
ping: bad address 'google.com'

# allow DNS traffic
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foo-deny-egress
spec:
  podSelector:
    matchLabels:
      app: foo
  policyTypes:
  - Egress
  egress:
  - ports:
    - port: 53
      protocol: TCP
    - port: 53
      protocol: UDP
EOF

# 可正常解析域名
$ kubectl run test-$RANDOM -it --rm --image=alpine --labels="app=foo" -- sh
/ # wget -qO- --timeout=2 http://web
wget: download timed out
/ # wget -qO- --timeout=2 http://google.com
wget: download timed out
/ # ping -W 2 -c 3 google.com
PING google.com (142.251.42.238): 56 data bytes
--- google.com ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

# 清理操作
kubectl delete pod,svc web
kubectl delete networkpolicy foo-deny-egress

5.12 DENY all non-whitelisted traffic from a namespace

Use Case: This is a fundamental policy, blocking all outgoing (egress) traffic from a namespace by default (including DNS resolution). After deploying this, you can deploy Network Policies that allow the specific outgoing traffic.

Consider applying this manifest to any namespace you deploy workloads to (except kube-system).

Best Practice: This policy will give you a default “deny all” functionality. This way, you can clearly identify which components have dependency on which components and deploy Network Policies which can be translated to dependency graphs between components.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-deny-all-egress
  namespace: default
spec:
  policyTypes:
  - Egress
  podSelector: {}     # empty, it will apply the policy to all pods of default namespace
  egress: []          # empty array, it causes all traffic (including DNS resolution) to be dropped

5.13 DENY external egress traffic

(a.k.a LIMIT traffic to pods in the cluster)

Use Cases:

• You want to prevent certain type of applications from establishing connections to the external networks.

$ kubectl run web --image=nginx --labels="app=web" --expose --port=80

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: foo-deny-external-egress
spec:
  podSelector:
    matchLabels:
      app: foo
  policyTypes:
  - Egress
  egress:
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
  - to:
    - namespaceSelector: {}  # empty, it will allow all pods in all namespaces, so the outbound traffic to pods in the cluster will be allowed
EOF

$ kubectl run test-$RANDOM -it --rm --image=alpine --labels="app=foo" -- sh
<!DOCTYPE html>
<html>
<head>
...

/ # wget -qO- --timeout=2 http://google.com
wget: download timed out

参考资料：

https://docs.nirmata.io/applicationmanagement/resources/networkpolicies/

https://github.com/ahmetb/kubernetes-network-policy-recipes