DPDK系列之十四：在kubernetes运行Intel的vhost-user-net-plugin插件以支持dpdk
一、前言
根据文章《容器云的数据通道加速方案概览》的描述，借助Intel提供的vhost-user-net-plugin CNI，可以为基于Kubernetes的容器配置使用virtio前端和vhost-user后端的数据通道加速方案，vhost-user后端port可以由OVS-DPDK或者VPP提供。
本文就将使用multus绑定flannel和vhost-user-net-plugin两种网络类型，为容器提供除了传统网络以外DPDK的支持。
multus的配置可以参照文章《kubernetes系列之十三：POD多网卡方案multus-cni之通过CRD给POD配置自由组合网络》。
转载自https://blog.csdn.net/cloudvtech
二、编译vhost-user-net-plugin CNI
2.1 参照git文档
github地址：https://github.com/intel/vhost-user-net-plugin
https://github.com/intel/vhost-user-net-plugin.git
git clone https://github.com/intel/vhost-user-net-plugin.git
cd vhost-user-net-plugin
./buildbuild出来的binary就在bin目录
[root@k8s-master vhost-user-net-plugin]# ls -l bin
总用量 3108
-rwxr-xr-x 1 root root 3179579 5月 29 07:56 vhostuser2.2 安装vhost-user-net-plugin CNI
将vhost-user-net-plugin目录下bin/vhostuser文件和tests目录拷贝到node的/opt/cni/bin目录
2.3 node上的CNI配置文件
本测试只使用一个master和一个node
rm -f /etc/cni/net.d/* 创建新的multus CRD conf如下：
/etc/cni/net.d/10-multus.conf
{
"name": "multus-cni-network",
"type": "multus",
"kubeconfig": "/etc/kubernetes/kubelet.conf"
} 重启kubelet
systemctl restart kubelet 转载自https://blog.csdn.net/cloudvtech
三、编译并运行OVS-DPDK
参见文章《DPD系列之一：DPDK 17.08.1在CentoS 7.2.1511的安装》和文章《DPDK系列之七：OVS-DPDK的在CentOS安装和测试》。
转载自https://blog.csdn.net/cloudvtech
四、编译容器镜像
参见文章《DPDK系列之十三：容器基于OVS-DPDK的dpdkvhostuser端口的pktgen和testpmd数据包测试》。
这里只要使用testpmd容器就可以了。
转载自https://blog.csdn.net/cloudvtech
五、部署vhost-user-net-plugin网络
5.1 multus Network CRD yml文件
ovs-dpdk-network.yaml
apiVersion: "kubernetes.com/v1"
kind: Network
metadata:
name: ovs-dpdk-networkobj
plugin: vhostuser
args: '[
{
"type": "vhostuser",
"name": "vhostuser-network",
"if0name": "net0",
"vhost": {
"vhost_tool": "/opt/cni/bin/tests/ovs-config.py"
},
"ipam": {
"type": "host-local",
"subnet": "192.168.166.0/24",
"rangeStart": "192.168.166.220",
"rangeEnd": "192.168.166.240",
"routes": [
{
"dst": "0.0.0.0/0"
}
],
"gateway": "192.168.166.2"
}
}
]' 使用vhost-user-net-plugin tests目录下ovs-config.py，这个tool的主要功能就是
使用vs-vsctl为POD创建一个dpdkvhostuser port并且将socket移动到/var/lib/cni/vhostuser/目录
5.2 部署vhost-user-net CNI网络
[root@k8s-master CDR]# kubectl get crd
NAME AGE
networks.kubernetes.com 21d
[root@k8s-master CDR]# kubectl get networks
NAME AGE
flannel-networkobj 21d
macvlan-networkobj 21d
ovs-dpdk-networkobj 3h
[root@k8s-master CDR]# kubectl describe networks ovs-dpdk-networkobj
Name: ovs-dpdk-networkobj
Namespace: default
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"kubernetes.com/v1","args":"[ { \"type\": \"vhostuser\", \"name\": \"vhostuser-network\", \"if0name\": \"net0\", \"vhost\": { \"vhost_too...
API Version: kubernetes.com/v1
Args: [ { "type": "vhostuser", "name": "vhostuser-network", "if0name": "net0", "vhost": { "vhost_tool": "/opt/cni/bin/tests/ovs-config.py" }, "ipam": { "type": "host-local", "subnet": "192.168.166.0/24", "rangeStart": "192.168.166.220", "rangeEnd": "192.168.166.240", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.166.2" } } ]
Kind: Network
Metadata:
Cluster Name:
Creation Timestamp: 2018-05-29T12:20:31Z
Generation: 1
Resource Version: 356234
Self Link: /apis/kubernetes.com/v1/namespaces/default/networks/ovs-dpdk-networkobj
UID: adfbbf3a-633a-11e8-8aca-000c29d3e746
Plugin: vhostuser
Events:
[root@k8s-master CDR]# 5.3 POD配置
cat testpmd_pod_flannel_vhostuser.yml
apiVersion: v1
kind: Pod
metadata:
name: nginx-flannel-vhostuser
annotations:
networks: '[
{ "name": "flannel-networkobj" },
{ "name": "ovs-dpdk-networkobj" }
]'
spec:
containers:
- name: ovs-dpdk
image: dpdk_testpmd_1708
imagePullPolicy: Never
securityContext:
privileged: true
command: [ "/bin/bash", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
volumeMounts:
- name: hugepage
mountPath: /mnt/huge
- name: socketdir
mountPath: /var/run/openvswitch
volumes:
- name: hugepage
hostPath:
path: /mnt/huge
- name: socketdir
hostPath:
path: /var/lib/cni/vhostuser部署两个POD
[root@k8s-master CDR]# kubectl apply -f testpmd_pod_flannel_vhostuser.yml
pod "nginx-flannel-vhostuser" created
[root@k8s-master CDR]# kubectl apply -f testpmd_pod_flannel_vhostuser1.yml
pod "nginx-flannel-vhostuser1" created转载自https://blog.csdn.net/cloudvtech
六、查看部署结果
6.1 log可以看到建立了两个port
May 29 12:12:57 k8s-node1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port br0 70dc0863789f-net0 -- set Interface 70dc0863789f-net0 type=dpdkvhostuser
May 29 12:12:59 k8s-node1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port br0 d7157cc4f590-net0 -- set Interface d7157cc4f590-net0 type=dpdkvhostuser2018-05-29T16:12:57.496Z|00185|netdev_dpdk|INFO|Socket /usr/local/var/run/openvswitch/70dc0863789f-net0 created for vhost-user port 70dc0863789f-net0
2018-05-29T16:12:57.496Z|00186|dpdk|INFO|VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/70dc0863789f-net0
2018-05-29T16:12:57.497Z|00187|dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 2 created.
2018-05-29T16:12:57.497Z|00188|dpif_netdev|INFO|There are 1 pmd threads on numa node 0
2018-05-29T16:12:57.578Z|00192|bridge|INFO|bridge br0: added interface 70dc0863789f-net0 on port 13
2018-05-29T16:12:59.288Z|00193|dpdk|INFO|VHOST_CONFIG: vhost-user server: socket created, fd: 51
2018-05-29T16:12:59.288Z|00194|netdev_dpdk|INFO|Socket /usr/local/var/run/openvswitch/d7157cc4f590-net0 created for vhost-user port d7157cc4f590-net0
2018-05-29T16:12:59.288Z|00195|dpdk|INFO|VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/d7157cc4f590-net0
2018-05-29T16:12:59.288Z|00196|bridge|INFO|bridge br0: added interface d7157cc4f590-net0 on port 146.2 POD状态
[root@k8s-master CDR]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-flannel-vhostuser 1/1 Running 0 10m 10.244.1.14 k8s-node1
nginx-flannel-vhostuser1 1/1 Running 0 10m 10.244.1.13 k8s-node1[root@k8s-master CDR]# ping 10.244.1.14
PING 10.244.1.14 (10.244.1.14) 56(84) bytes of data.
64 bytes from 10.244.1.14: icmp_seq=1 ttl=63 time=0.504 ms
64 bytes from 10.244.1.14: icmp_seq=2 ttl=63 time=0.466 ms
64 bytes from 10.244.1.14: icmp_seq=3 ttl=63 time=0.512 ms
^C
--- 10.244.1.14 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.466/0.494/0.512/0.020 ms[root@k8s-node1 ~]# docker ps | grep bash
4663c3f744ff 284b78f4e48f "/bin/bash -c -- '..." 3 minutes ago Up 3 minutes k8s_ovs-dpdk_nginx-flannel-vhostuser_default_31243b57-635a-11e8-8aca-000c29d3e746_0
668df5db09c6 284b78f4e48f "/bin/bash -c -- '..." 3 minutes ago Up 3 minutes k8s_ovs-dpdk_nginx-flannel-vhostuser1_default_2fefba60-635a-11e8-8aca-000c29d3e746_0[root@k8s-node1 ~]# docker exec -it 4663c3f744ff bash
[root@nginx-flannel-vhostuser dpdk-17.05]# ifconfig
eth0: flags=4163 mtu 1450
inet 10.244.1.14 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::c25:caff:fe9d:5b8d prefixlen 64 scopeid 0x20
ether 0a:58:0a:f4:01:0e txqueuelen 0 (Ethernet)
RX packets 13 bytes 962 (962.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
6.4 vhost socket
POD的网络容器pause状态如下：
d7157cc4f590 k8s.gcr.io/pause-amd64:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_nginx-flannel-vhostuser_default_31243b57-635a-11e8-8aca-000c29d3e746_0
70dc0863789f k8s.gcr.io/pause-amd64:3.1 "/pause" 3 minutes ago Up 3 minutes k8s_POD_nginx-flannel-vhostuser1_default_2fefba60-635a-11e8-8aca-000c29d3e746_0所以POD的vhost socket应该在/var/lib/cni/vhostuser目录，对应两个pause容器ID
[root@k8s-node1 ~]# ls -l /var/lib/cni/vhostuser/
总用量 0
drwx------ 2 root root 95 5月 29 12:12 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a
drwx------ 2 root root 95 5月 29 12:12 d7157cc4f590a84c0e460daf8062b44a1c4b80d4ec6213551915dd35527caf68[root@k8s-node1 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a]# ls -l
总用量 8
srwxr-xr-x 1 root root 0 5月 29 12:12 70dc0863789f-net0
-rw-r--r-- 1 root root 127 5月 29 12:12 70dc0863789f-net0-ip4.conf
-rw-r--r-- 1 root root 156 5月 29 12:12 70dc0863789f-net0.json[root@k8s-node1 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a]# cat 70dc0863789f-net0-ip4.conf
{
"ipAddr": "192.168.166.228/32",
"macAddr": "8e:10:52:bf:e5:70",
"gateway": "169.254.1.1",
"gwMac": "00:00:00:00:00:00"
}
[root@k8s-node1 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a]# cat 70dc0863789f-net0.json
{"vhostname":"70dc0863789f-net0","vhostmac":"00:00:00:00:00:00","ifname":"net0","ifmac":"8e:10:52:bf:e5:70","vhost_tool":"/opt/cni/bin/tests/ovs-config.py"}6.5 OVS-DPDK port
/root/openvswitch-2.8.1/utilities/ovs-vsctl show
590c0cb0-706c-4d36-a9ed-611a824a5a9e
Bridge "br0"
Port "70dc0863789f-net0"
Interface "70dc0863789f-net0"
type: dpdkvhostuser
Port "d7157cc4f590-net0"
Interface "d7157cc4f590-net0"
type: dpdkvhostuser
Port "br0"
Interface "br0"
type: internal