Cilium Host Routing Mode

Cilium ConfigMap

cilium configmap 是配置cilium的地方：

root@node1:~# kubectl get cm -n kube-system | grep cilium
cilium-config                        34     7d21h
root@node1:~# kubectl edit cm -n kube-system cilium-config
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  agent-health-port: "9879"
  auto-direct-node-routes: "False"
  bpf-ct-global-any-max: "262144"
  bpf-ct-global-tcp-max: "524288"
  bpf-map-dynamic-size-ratio: "0.0025"
  cgroup-root: /run/cilium/cgroupv2
  clean-cilium-bpf-state: "false"
  clean-cilium-state: "false"
  cluster-name: default
  custom-cni-conf: "false"
  debug: "True"
  disable-cnp-status-updates: "True"
  enable-bpf-clock-probe: "True"
  enable-bpf-masquerade: "True"
  enable-host-legacy-routing: "False"
  enable-ip-masq-agent: "False"
  enable-ipv4: "True"
  enable-ipv4-masquerade: "True"
  enable-ipv6: "False"
  enable-ipv6-masquerade: "True"
  enable-remote-node-identity: "True"
  enable-well-known-identities: "False"
  etcd-config: |-
    ---
    endpoints:
      - https://192.168.64.8:2379
      - https://192.168.64.9:2379
      - https://192.168.64.10:2379

    # In case you want to use TLS in etcd, uncomment the 'ca-file' line
    # and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    ca-file: "/etc/cilium/certs/ca_cert.crt"

    # In case you want client to server authentication, uncomment the following
    # lines and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    key-file: "/etc/cilium/certs/key.pem"
    cert-file: "/etc/cilium/certs/cert.crt"
  identity-allocation-mode: kvstore
  ipam: kubernetes
  kube-proxy-replacement: probe
  kvstore: etcd
  kvstore-opt: '{"etcd.config": "/var/lib/etcd-config/etcd.config"}'
  monitor-aggregation: medium
  monitor-aggregation-flags: all
  operator-api-serve-addr: 127.0.0.1:9234
  preallocate-bpf-maps: "False"
  sidecar-istio-proxy-image: cilium/istio_proxy
  tunnel: vxlan
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"agent-health-port":"9879","auto-direct-node-routes":"False","bpf-ct-global-any-max":"262144","bpf-ct-global-tcp-max":"524288","bpf-map-dynamic-size-ratio":"0.0025","cgroup-root":"/run/cilium/cgroupv2","clean-cilium-bpf-state":"false","clean-cilium-state":"false","cluster-name":"default","custom-cni-conf":"false","debug":"False","disable-cnp-status-updates":"True","enable-bpf-clock-probe":"True","enable-bpf-masquerade":"False","enable-host-legacy-routing":"True","enable-ip-masq-agent":"False","enable-ipv4":"True","enable-ipv4-masquerade":"True","enable-ipv6":"False","enable-ipv6-masquerade":"True","enable-remote-node-identity":"True","enable-well-known-identities":"False","etcd-config":"---\nendpoints:\n  - https://192.168.64.8:2379\n  - https://192.168.64.9:2379\n  - https://192.168.64.10:2379\n\n# In case you want to use TLS in etcd, uncomment the 'ca-file' line\n# and create a kubernetes secret by following the tutorial in\n# https://cilium.link/etcd-config\nca-file: \"/etc/cilium/certs/ca_cert.crt\"\n\n# In case you want client to server authentication, uncomment the following\n# lines and create a kubernetes secret by following the tutorial in\n# https://cilium.link/etcd-config\nkey-file: \"/etc/cilium/certs/key.pem\"\ncert-file: \"/etc/cilium/certs/cert.crt\"","identity-allocation-mode":"kvstore","ipam":"kubernetes","kube-proxy-replacement":"probe","kvstore":"etcd","kvstore-opt":"{\"etcd.config\": \"/var/lib/etcd-config/etcd.config\"}","monitor-aggregation":"medium","monitor-aggregation-flags":"all","operator-api-serve-addr":"127.0.0.1:9234","preallocate-bpf-maps":"False","sidecar-istio-proxy-image":"cilium/istio_proxy","tunnel":"vxlan"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"cilium-config","namespace":"kube-system"}}
  creationTimestamp: "2023-07-08T12:42:52Z"
  name: cilium-config
  namespace: kube-system
  resourceVersion: "168401"
  uid: 0fbe4759-a8c4-457b-a06f-5fc6231272b9

具体每个字段的意思可参考：Cilium 配置详解

如果有需要修改的修改保存退出后需要重启cilium-agent生效：

root@node1:~# kubectl rollout restart ds/cilium -n kube-system

重启后进入任意一个cilium-agent容器中：

root@node1:/home/cilium# cilium status
KVStore:                 Ok   etcd: 3/3 connected, lease-ID=797e89450e0683a2, lock lease-ID=797e89450e0b2b6a, has-quorum=true: https://192.168.64.10:2379 - 3.5.4; https://192.168.64.8:2379 - 3.5.4 (Leader); https://192.168.64.9:2379 - 3.5.4
Kubernetes:              Ok   1.24 (v1.24.6) [linux/amd64]
Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumNetworkPolicy", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:    Probe   [enp0s2 192.168.64.8]
Host firewall:           Disabled
CNI Chaining:            none
Cilium:                  Ok   1.12.1 (v1.12.1-4c9a630)
NodeMonitor:             Disabled
Cilium health daemon:    Ok   
IPAM:                    IPv4: 5/254 allocated from 10.233.64.0/24, 
BandwidthManager:        Disabled
Host Routing:            BPF # 注意这里
Masquerading:            BPF   [enp0s2]   10.233.64.0/24 [IPv4: Enabled, IPv6: Disabled] # 表明只是在enp0s2接口上进行NAT的接口。[两种模式，一种是BPF。一种是在Legacy Mode下使用IPtables]
Controller Status:       43/43 healthy
Proxy Status:            OK, ip 10.233.64.93, 0 redirects active on ports 10000-20000
Global Identity Range:   min 256, max 65535
Hubble:                  Disabled
Encryption:              Disabled
Cluster health:          3/3 reachable   (2023-07-16T10:01:41Z)

Host Routing

即使 Cilium 使用 eBPF 执行网络路由，默认情况下网络数据包仍会遍历节点的常规网络堆栈的某些部分。 这样可以确保所有数据包仍然遍历所有 iptables hooks，以防你依赖它们。然而，这样会增加大量开销。
在 Cilium 1.9 中引入了基于 eBPF 的​​ ​host-routing​​，来完全绕过 iptables 和上层主机堆栈，与常规 veth 设备操作相比，实现了更快的网络命名空间切换。 如果您的内核支持，此选项会自动启用。

eBPF host-Routing Legacy

Legacy是传统的eBPF host-Routing 模式，左侧是标准的网络流向过程，可以发现iptables 会对包进行处理。

eBPF host-Routing BPF

右侧是启用ebpf host-routing, 直接跨过iptables，必要条件：

 Kernel >= 5.10
 Direct-routing or tunneling
 eBPF-based 替换kube-proxy
 eBPF-based masquerading

此模式由于引入了两个helper()函数加持了eBPF host-Routing，那么在DataPath上就会有不同的地方：

• bpf_redirect_peer()
  简单描述：
  同节点的Pod通信时候，Pod-A从自己的veth pair 的lxc出来以后直接送到Pod-B的veth pair的eth0网卡，而绕过Pod-B的veth pair的lxc网卡。

• bpf_redirect_neigh()
  简单描述：
  不同节点Pod通信的时候，Pod-A从自己的veth pair的lxc出来以后被自己的ingress方向上的from-container处理（vxlan封装或是native routing）然后tail call到vxlan处理然后再走bpf_redirect_neigh()到物理网卡，然后到对端Pod-B所在的Node节点，此时再进行反向处理即可。这里有一点注意：就是from-container —> vxlan 封装 —> bpf_redirect_neigh () 处理。