社区实践 | KubeVirt通过Kube-OVN接入Underlay网络

本篇文章转自Kube-OVN社区成员技术博客，介绍了KubeVirt通过Kube-OVN的Underlay网络模式，使虚拟机接入物理网络的实现。

作者：苏楷，宁算（南京）科技有限公司高级工程师

实践操作

检查Kubernetes使用的Pod网络地址段和Service网络地址段。

sukai@ubuntuserver:/etc/kubernetes/manifests$ sudo grep cidr kube-controller-manager.yaml
    - --allocate-node-cidrs=true
    - --cluster-cidr=10.244.0.0/16
sukai@ubuntuserver:/etc/kubernetes/manifests$ sudo grep range kube-controller-manager.yaml
    - --service-cluster-ip-range=10.211.0.0/16

检查kube-ovn-controller配置项是否正确

kube-ovn在版本v1.11.0开始支持keep-vm-ip参数，虚拟机使用固定IP地址。

Containers:
  kube-ovn-controller:
    Container ID:  containerd://0f8d53403c9821c7b535c64d2c98e1ca130a15e1b2b878270cd9388c36d0e48a
    Image:         kubeovn/kube-ovn:v1.11.0
    Image ID:      docker.io/kubeovn/kube-ovn@sha256:fea623e68a2a81ef78102c6cfe95b96c16c054c57f4c8b9d168bd3ff620b6779
    Port:          
    Host Port:     
    Args:
      /kube-ovn/start-controller.sh
      --default-cidr=10.244.0.0/16
      --default-gateway=10.244.0.1
      --default-gateway-check=true
      --default-logical-gateway=false
      --default-exclude-ips=
      --node-switch-cidr=100.64.0.0/16
      --service-cluster-ip-range=10.211.0.0/16
      --network-type=geneve
      --default-interface-name=
      --default-vlan-id=100
      --pod-nic-type=veth-pair
      --enable-lb=true
      --enable-np=true
      --enable-eip-snat=true
      --enable-external-vpc=true
      --keep-vm-ip=true

创建服务网络

指定物理服务器上访问物理网络的网卡。

apiVersion: kubeovn.io/v1
kind: ProviderNetwork
metadata:
  name: office
spec:
  defaultInterface: eno1

创建VLAN

创建在指定服务网络上创建一个VLAN，这里VLAN ID为0表示不属于任何VLAN。

apiVersion: kubeovn.io/v1
kind: Vlan
metadata:
  name: defaultvlan
spec:
  id: 0
  provider: office

创建子网

在defaultvlan上，创建一个子网，这个子网作用的namespace为mail263，指定子网分配的物理网络地址段，物理网络网关地址。

---
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: mail263
spec:
  protocol: IPv4
  cidrBlock: 172.16.3.0/24
  gateway: 172.16.3.2
  excludeIps:
  - 172.16.3.1..172.16.3.200
  vlan: defaultvlan
  #  natOutgoing: false
  #private: false
  namespaces:
  - mail263

查看kube-ovn信息

sukai@ubuntuserver:~$ kubectl get vlan -o wide
NAME          ID    PROVIDER
defaultvlan   0     office
sukai@ubuntuserver:~$ kubectl get subnet
NAME          PROVIDER   VPC           PROTOCOL   CIDR            PRIVATE   NAT     DEFAULT   GATEWAYTYPE   V4USED   V4AVAILABLE   V6USED   V6AVAILABLE   EXCLUDEIPS
join          ovn        ovn-cluster   IPv4       100.64.0.0/16   false     false   false     distributed   1        65532         0        0             ["100.64.0.1"]
mail263       ovn        ovn-cluster   IPv4       172.16.3.0/24   false     false   false     distributed   1        53            0        0             ["172.16.3.1..172.16.3.200"]
ovn-default   ovn        ovn-cluster   IPv4       10.244.0.0/16   false     true    true      distributed   19       65514         0        0             ["10.244.0.1"]
sukai@ubuntuserver:~$

sukai@ubuntuserver:~$ kubectl ko nbctl show
switch e0b180e4-1e4c-49e0-9bea-d5aa384196b0 (ovn-default)
    port cert-manager-8568b6f9cb-wf74c.cert-manager
        addresses: ["00:00:00:E9:A4:E5 10.244.0.15"]
    port cert-manager-cainjector-c9c77b797-xwlgh.cert-manager
        addresses: ["00:00:00:39:46:D0 10.244.0.14"]
    port coredns-6d4b75cb6d-tt7cb.kube-system
        addresses: ["00:00:00:F1:29:07 10.244.0.3"]
    port virt-handler-9st4d.kubevirt
        addresses: ["00:00:00:A0:72:A1 10.244.0.22"]
    port hostpath-provisioner-csi-5mcbc.hostpath-provisioner
        addresses: ["00:00:00:D0:28:68 10.244.0.18"]
    port virt-api-748cfcdb48-vz6fk.kubevirt
        addresses: ["00:00:00:D8:BF:D4 10.244.0.20"]
    port nginx.default
        addresses: ["00:00:00:F3:E4:1C 10.244.0.5"]
    port virt-controller-5756d69f5f-ffs5r.kubevirt
        addresses: ["00:00:00:12:42:FC 10.244.0.23"]
    port cdi-deployment-7d587548cd-w55fh.cdi
        addresses: ["00:00:00:D3:7E:DF 10.244.0.34"]
    port cdi-uploadproxy-6c64d5cd59-5lvkc.cdi
        addresses: ["00:00:00:B2:C7:B7 10.244.0.35"]
    port virt-operator-5fcd4ff76f-j6x24.kubevirt
        addresses: ["00:00:00:5A:C9:3E 10.244.0.7"]
    port cdi-apiserver-c6cdc9489-g57k4.cdi
        addresses: ["00:00:00:11:AF:5B 10.244.0.33"]
    port virt-controller-5756d69f5f-lmkpk.kubevirt
        addresses: ["00:00:00:CE:9F:04 10.244.0.21"]
    port virt-operator-5fcd4ff76f-h8sx6.kubevirt
        addresses: ["00:00:00:C2:56:F1 10.244.0.6"]
    port cert-manager-webhook-885b8ffcb-hrzjt.cert-manager
        addresses: ["00:00:00:11:A1:8B 10.244.0.16"]
    port hostpath-provisioner-operator-7c8c499c77-46vk8.hostpath-provisioner
        addresses: ["00:00:00:1D:58:B7 10.244.0.17"]
    port coredns-6d4b75cb6d-l6cjw.kube-system
        addresses: ["00:00:00:A6:76:70 10.244.0.2"]
    port kube-ovn-pinger-tb5t7.kube-system
        addresses: ["00:00:00:A7:BB:47 10.244.0.4"]
    port cdi-operator-997bdf879-ht6kx.cdi
        addresses: ["00:00:00:DD:1E:29 10.244.0.32"]
    port ovn-default-ovn-cluster
        type: router
        router-port: ovn-cluster-ovn-default
switch 0629dbe7-beab-45f1-8d42-0ce2f2a89a79 (mail263)
    port localnet.mail263
        type: localnet
        addresses: ["unknown"]
    port mailserver.mail263
        addresses: ["00:00:00:68:56:C6 172.16.3.250"]
switch 73e8db33-6e71-4f1e-af8a-516afaf13019 (join)
    port node-ubuntuserver
        addresses: ["00:00:00:B0:35:E1 100.64.0.2"]
    port join-ovn-cluster
        type: router
        router-port: ovn-cluster-join
router e2c95d29-f93b-4f70-95b7-0f1c5e3c982a (ovn-cluster)
    port ovn-cluster-join
        mac: "00:00:00:E2:21:46"
        networks: ["100.64.0.1/16"]
    port ovn-cluster-ovn-default
        mac: "00:00:00:C9:39:03"
        networks: ["10.244.0.1/16"]
sukai@ubuntuserver:~$

创建虚拟机

因为我们需要一个CentOS 6.8操作系统，这个古老的系统因为驱动原因无法在现在的服务器硬件上安装，所以我们通过虚拟机来解决这个问题。当然使用虚拟化也需要解决一些新的虚拟化技术兼容CentOS 6.8的问题。

上传虚拟机镜像

sukai@ubuntuserver:/etc/kubernetes/manifests$ wget https://cloud.centos.org/centos/6/images/CentOS-6-x86_64-GenericCloud-1605.qcow2
sukai@ubuntuserver:/etc/kubernetes/manifests$ kubectl -n cdi get svc
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
cdi-api                  ClusterIP   10.211.69.105           443/TCP    20h
cdi-prometheus-metrics   ClusterIP   10.211.27.150           8080/TCP   20h
cdi-uploadproxy          ClusterIP   10.211.41.23            443/TCP    20h
sukai@ubuntuserver:/etc/kubernetes/manifests$ virtctl image-upload dv centos68 -vm-disk --size=10Gi -n mail263 --storage-class hostpath-csi --image-path=/home/sukai/CentOS-6-x86_64-GenericCloud-1907.qcow2 --uploadproxy-url=https://10.211.41.23 --insecure  --wait-secs=240

创建虚拟机磁盘的PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: centos-mail263-disk
  namespace: mail263
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: hostpath-csi
  resources:
    requests:
      storage: 100Gi

创建虚拟机

1，CPU model使用host-passthrough，不然会性能告警提示：Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only.

2，disk的bus磁盘设备类型使用sata

3，网卡设备类型使用e1000

4，初始化配置用户密码

---
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: mailserver
  name: mailserver
  namespace: mail263
spec:
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/vm: mailserver
    spec:
      domain:
        cpu:
          model: host-passthrough
        devices:
          disks:
          - name: osdisk
            disk:
              bus: sata
          - disk:
              bus: sata
            name: cloudinitdisk
          - name: datadisk
            disk:
              bus: sata
          interfaces:
          - name: default
            model: e1000
            bridge: {}
        resources:
          requests:
            cpu: "8"
            memory: 16Gi
      networks:
      - name: default
        pod: {}
      terminationGracePeriodSeconds: 0
      volumes:
      - name: osdisk
        dataVolume:
          name: centos68
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            user: centos
            password: password
            ssh_pwauth: True
            chpasswd: { expire: False }
      - name: datadisk
        persistentVolumeClaim:
          claimName: centos-mail263-disk

启动虚拟机

sukai@ubuntuserver:/etc/kubernetes/manifests$ virtctl -n mail263 start mailserver
sukai@ubuntuserver:/etc/kubernetes/manifests$ kubectl -n mail263 get vms
NAME         AGE     STATUS    READY
mailserver   4h18m   Running   True
sukai@ubuntuserver:/etc/kubernetes/manifests$ kubectl -n mail263 get vmi
NAME         AGE    PHASE     IP             NODENAME       READY
mailserver   133m   Running   172.16.3.250   ubuntuserver   True
sukai@ubuntuserver:/etc/kubernetes/manifests$

Console登录虚拟机

sukai@ubuntuserver:/etc/kubernetes/manifests$ virtctl console -n mail263 mailserver
Successfully connected to mailserver console. The escape sequence is ^]

login: root
Password:
Last login: Thu Jun  9 06:31:57 from 172.16.3.25
[root@mailserver ~]# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:00:00:68:56:c6 brd ff:ff:ff:ff:ff:ff
    inet 172.16.3.250/24 brd 172.16.3.255 scope global eth0
    inet 192.168.10.95/24 brd 192.168.10.255 scope global eth0:0
    inet6 fe80::200:ff:fe68:56c6/64 scope link
       valid_lft forever preferred_lft forever
[root@mailserver ~]# ip route
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.250
192.168.10.0/24 dev eth0  proto kernel  scope link  src 192.168.10.95
10.65.0.0/16 via 192.168.10.99 dev eth0
default via 172.16.3.2 dev eth0  proto static
[root@mailserver ~]# ping www.sina.com
PING ww1.sinaimg.cn.w.alikunlun.com (163.181.42.223) 56(84) bytes of data.
64 bytes from 163.181.42.223: icmp_seq=1 ttl=55 time=75.5 ms

--- ww1.sinaimg.cn.w.alikunlun.com ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1018ms
rtt min/avg/max/mdev = 75.505/75.505/75.505/0.000 ms
[root@mailserver ~]# ping www.sina.com -c 1
PING ww1.sinaimg.cn.w.alikunlun.com (163.181.42.223) 56(84) bytes of data.
64 bytes from 163.181.42.223: icmp_seq=1 ttl=55 time=75.1 ms

--- ww1.sinaimg.cn.w.alikunlun.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 78ms
rtt min/avg/max/mdev = 75.104/75.104/75.104/0.000 ms
[root@mailserver ~]# sukai@ubuntuserver:/etc/kubernetes/manifests$
sukai@ubuntuserver:/etc/kubernetes/manifests$

容器与虚拟机的网络通信

sukai@ubuntuserver:~$ kubectl exec -it nginx -- /bin/sh
/ # ping 172.16.3.250 -c1
PING 172.16.3.250 (172.16.3.250): 56 data bytes
64 bytes from 172.16.3.250: seq=0 ttl=62 time=0.586 ms

--- 172.16.3.250 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.586/0.586/0.586 ms
/ # ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
18: eth0@if19:  mtu 1400 qdisc noqueue state UP
    link/ether 00:00:00:f3:e4:1c brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.5/16 brd 10.244.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::200:ff:fef3:e41c/64 scope link
       valid_lft forever preferred_lft forever
/ # ip r
default via 10.244.0.1 dev eth0
10.244.0.0/16 dev eth0 scope link  src 10.244.0.5
/ #
sukai@ubuntuserver:~$

注意

Kube-OVN在使用Underlay网络模式或者在自定义VPC中，都需要Kubernetes节点主机配置相应的网络安全策略，比如PortSecurity, 允许混杂模式，允许MAC更改，允许伪传输等。在使用VMware和Openstack虚拟机部署Kubernetes集群时，尤其需要注意。

总结

Kube-OVN提供了Underlay网络模式，在此模式下，提供了容器和虚拟机接入物理网络的能力，并且能够让容器与虚拟机在默认的VPC下进行通信。