k8s mellanox网卡使用dpdk驱动问题总结

本文主要总结一下在k8s环境中，mellanox网卡使用dpdk driver可能会遇到的问题及解决办法。

1. 不能挂载 /sys 目录到pod中
其他厂家的网卡，比如intel的x710等，如果想在k8s中，使用dpdk driver，/sys目录是必须挂载的，因为dpdk启动过程会读取这个目录下的文件。但是对于mellanox网卡来说，它是比较特殊的，在使用dpdk driver时，也必须绑定在kernel driver mlx5_core上面。

如果挂载了 /sys 目录到pod中，就会报如下的错误。

net_mlx5: port 0 cannot get MAC address, is mlx5_en loaded? (errno: No such file or directory)
net_mlx5: probe of PCI device 0000:00:09.0 aborted after encountering an error: No such device
EAL: Requested device 0000:00:09.0 cannot be used

原因是 host上的 /sys/ 会覆盖 pod 里的 /sys/ 内容，而 mlx 网卡会读取这些目录，比如 /sys/devices/pci0000:00/0000:00:09.0/net/，如果覆盖了，就会报错。下面分析下代码

mlx5_pci_probe
    mlx5_dev_spawn
        /* Configure the first MAC address by default. */
        if (mlx5_get_mac(eth_dev, &mac.addr_bytes)) {
            DRV_LOG(ERR,
                "port %u cannot get MAC address, is mlx5_en"
                " loaded? (errno: %s)",
                eth_dev->data->port_id, strerror(rte_errno));
            err = ENODEV;
            goto error;
        }

//如果host上的 /sys/ 覆盖 pod 里的 /sys/ 内容，会出现问题(具体哪行代码出问题还有待调查)
int
mlx5_get_mac(struct rte_eth_dev *dev, uint8_t (*mac)[ETHER_ADDR_LEN])
{
    struct ifreq request;
    int ret;

    ret = mlx5_ifreq(dev, SIOCGIFHWADDR, &request);
        int sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_IP);
        mlx5_get_ifname(dev, &ifr->ifr_name);
        ioctl(sock, req, ifr);
                
    if (ret)
        return ret;
    memcpy(mac, request.ifr_hwaddr.sa_data, ETHER_ADDR_LEN);
    return 0;
}

2. dpdk启动过程中找不到mellanox网卡
有时候会出现下面的错误，找不到mellanox网卡，并且提示了是不是没有加载kernel driver，如果没有在k8s环境中，这个提示是有用的，可能网卡确实没有绑定到mlx5_core。

EAL: PCI device 0000:00:06.0 on NUMA socket -1"}
EAL:   Invalid NUMA socket, default to 0"}
EAL:   probe driver: 15b3:101a net_mlx5"}
net_mlx5: no Verbs device matches PCI device 0000:00:06.0, are kernel drivers loaded?"}
EAL: Requested device 0000:00:06.0 cannot be used"}

但是在k8s环境中，就有其他原因了。问题现象是，如果启动的pod有privileged权限，那dpdk是可以识别到网卡并且启动成功的，但是如果没有privileged权限，就会报上面的错误。
先分析下为什么报这个错误，dpdk代码如下

mlx5_pci_probe
    unsigned int n = 0;
    //调用 libibverbs 里的函数 ibv_get_device_list 获取 ibv 设备
    ibv_list = mlx5_glue->get_device_list(&ret);
    while (ret-- > 0) {
        ibv_match[n++] = ibv_list[ret];
    }
    //在这里报错，说明n为0，n在上面遍历ret的时候赋值，n为0，说明ret也为0
    if (!n) {
        DRV_LOG(WARNING,
            "no Verbs device matches PCI device " PCI_PRI_FMT ","
            " are kernel drivers loaded?",
            pci_dev->addr.domain, pci_dev->addr.bus,
            pci_dev->addr.devid, pci_dev->addr.function);
        rte_errno = ENOENT;
        ret = -rte_errno;
    }

要想知道为什么ret也为0，得分析下 libibverbs 源码，libibverbs 源码可以在 OFED 包中找到，路径如下 MLNX_OFED_LINUX-5.2-2.2.0.0-ubuntu16.04-x86_64/src/MLNX_OFED_SRC-5.2-2.2.0.0/SOURCES/rdma-core-52mlnx1/libibverbs

LATEST_SYMVER_FUNC(ibv_get_device_list, 1_1, "IBVERBS_1.1",
           struct ibv_device **,
           int *num)
    ibverbs_get_device_list(&device_list);
        find_sysfs_devs(&sysfs_list);
            setup_sysfs_dev
                try_access_device(sysfs_dev)
                    struct stat cdev_stat;
                    char *devpath;
                    int ret;
                    //查看这个文件是否存在 /dev/infiniband/uverbs0
                    if (asprintf(&devpath, RDMA_CDEV_DIR"/%s", sysfs_dev->sysfs_name) < 0)
                        return ENOMEM;

                    ret = stat(devpath, &cdev_stat);
                    free(devpath);
                    return ret;

由上面代码可知，ibverbs会检查/dev/infiniband/uverbs0文件是否存在(一个网卡对应一个uverbs0)，如果不存在就认为没有找到网卡。
如果pod有privileged权限，pod内就可以读取/dev/infiniband/uverbs0这些文件，如果没有privileged权限，pod内部是没有/dev/infiniband这个目录的。

//pod没有privilege特权时
root@pod-dpdk:~/dpdk/x86_64-native-linuxapp-gcc/app# ls /dev
core  fd  full  mqueue  null  ptmx  pts  random  shm  stderr  stdin  stdout  termination-log  tty  urandom  zero

//pod有privilege特权时，可看到infiniband
root@pod-dpdk:~/# ls /dev/
autofs           infiniband        mqueue              sda2             tty12  tty28  tty43  tty59   ttyS16  ttyS31     vcs4   vfio
...

root@pod-dpdk:~# ls /dev/infiniband/
uverbs0

由上可知，pod里有没有这个文件 /dev/infiniband/uverbs0 是关键。如果pod内没有这些 /dev/infiniband 文件就会报上面的错误。

那这个文件怎么才能传到pod里呢？有两种办法
a. 给pod privileged权限，这种方式是不需要将 /dev/ 挂载到pod内部的，因为有特权的pod可以直接访问host上的这些文件
b. 使用 k8s 的 sriov-network-device-plugin，要注意版本，高点的版本才能避免此问题。如果是mlx网卡，它会将需要的device通过 docker的 --device 挂载给container，比如下面的环境，将
/dev/infiniband/uverbs2等文件挂载到container中

root# docker inspect 1dfe96c8eff4
[
    {
        "Id": "1dfe96c8eff4c8ede0d8eb4e480fec9f002f68c4da1bb5265580ee968c6d7502",
        "Created": "2021-04-12T03:24:22.598030845Z",
        ...
        "HostConfig": {
            ...
            "CapAdd": [
                "NET_RAW",
                "NET_ADMIN",
                "IPC_LOCK"
            ],

            "Privileged": false,

            "Devices": [
                {
                    "PathOnHost": "/dev/infiniband/ucm2",
                    "PathInContainer": "/dev/infiniband/ucm2",
                    "CgroupPermissions": "rwm"
                },
                {
                    "PathOnHost": "/dev/infiniband/issm2",
                    "PathInContainer": "/dev/infiniband/issm2",
                    "CgroupPermissions": "rwm"
                },
                {
                    "PathOnHost": "/dev/infiniband/umad2",
                    "PathInContainer": "/dev/infiniband/umad2",
                    "CgroupPermissions": "rwm"
                },
                {
                    "PathOnHost": "/dev/infiniband/uverbs2",
                    "PathInContainer": "/dev/infiniband/uverbs2",
                    "CgroupPermissions": "rwm"
                },
                {
                    "PathOnHost": "/dev/infiniband/rdma_cm",
                    "PathInContainer": "/dev/infiniband/rdma_cm",
                    "CgroupPermissions": "rwm"
                }
            ],
            ...
        },

sriov plguin 的 代码，将hostpath上的目录挂载到pod中

// NewRdmaSpec returns the RdmaSpec
func NewRdmaSpec(pciAddrs string) types.RdmaSpec {
        deviceSpec := make([]*pluginapi.DeviceSpec, 0)
        isSupportRdma := false
        rdmaResources := rdmamap.GetRdmaDevicesForPcidev(pciAddrs)
        if len(rdmaResources) > 0 {
                isSupportRdma = true
                for _, res := range rdmaResources {
                        resRdmaDevices := rdmamap.GetRdmaCharDevices(res)
                        for _, rdmaDevice := range resRdmaDevices {
                                deviceSpec = append(deviceSpec, &pluginapi.DeviceSpec{
                                        HostPath:      rdmaDevice,  
                                        ContainerPath: rdmaDevice,
                                        Permissions:   "rwm",
                                })
                        }
                }
        }

        return &rdmaSpec{isSupportRdma: isSupportRdma, deviceSpec: deviceSpec}
}

sriov plguin 的 log，可看到它会把 /dev/infiniband 下面的几个文件传到pod

###/var/log/sriovdp/sriovdp.INFO
I0412 03:17:57.120886    5327 server.go:123] AllocateResponse send: &AllocateResponse{ContainerResponses:[]
*ContainerAllocateResponse{&ContainerAllocateResponse{Envs:map[string]string{PCIDEVICE_INTEL_COM_DP_SRIOV_MLX5: 0000:00:0a.0,},
Mounts:[]*Mount{},Devices:[]*DeviceSpec{&DeviceSpec{ContainerPath:/dev/infiniband/ucm2,HostPath:/dev/infiniband/ucm2,Permissions:rwm,},
&DeviceSpec{ContainerPath:/dev/infiniband/issm2,HostPath:/dev/infiniband/issm2,Permissions:rwm,},
&DeviceSpec{ContainerPath:/dev/infiniband/umad2,HostPath:/dev/infiniband/umad2,Permissions:rwm,},
&DeviceSpec{ContainerPath:/dev/infiniband/uverbs2,HostPath:/dev/infiniband/uverbs2,Permissions:rwm,},
&DeviceSpec{ContainerPath:/dev/infiniband/rdma_cm,HostPath:/dev/infiniband/rdma_cm,Permissions:rwm,},},
Annotations:map[string]string{},},},}

3. 没有特权，dpdk启动失败
为了安全考虑，不会给pod特权，这样在pod内部启动dpdk会失败，报错如下

root@pod-dpdk:~/dpdk/x86_64-native-linuxapp-gcc/app# ./l2fwd -cf -n4  -w 00:09.0 -- -p1
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: Cannot obtain physical addresses: No such file or directory. Only vfio will function.
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
error allocating rte services array

dpdk的启动参数加上参数 -iova-mode=va，这样就不需要特权了

root@pod-dpdk:~/dpdk/x86_64-native-linuxapp-gcc/app# ./l2fwd -cf -n4  -w 00:09.0 --iova-mode=va -- -p1
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:00:09.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15b3:1016 net_mlx5

4. 读取网卡统计计数失败

dpdk提供了两个函数，用来获取网卡计数，rte_eth_stats_get和rte_eth_xstats_get，前者用来获取几个固定的计数，比如收发报文数和字节数，
后者用来获取扩展计数，每种网卡都有它自己的计数。

对于mellanox网卡来说，它的dpdk driver mlx5也提供了对应的函数，如下

rte_eth_stats_get -> stats_get -> mlx5_stats_get 
rte_eth_xstats_get -> xstats_get -> mlx5_xstats_get

mlx5_stats_get 在pod内部可以正常读取计数，但是mlx5_xstats_get就会遇到问题。
mlx5_xstats_get -> mlx5_read_dev_counters 这个函数会读取下面路径的文件获取计数

root# ls /sys/devices/pci0000\:00/0000\:00\:09.0/infiniband/mlx5_0/ports/1/hw_counters/
duplicate_request      out_of_buffer    req_cqe_flush_error         resp_cqe_flush_error       rx_atomic_requests
implied_nak_seq_err    out_of_sequence  req_remote_access_errors    resp_local_length_error    rx_read_requests
lifespan               packet_seq_err   req_remote_invalid_request  resp_remote_access_errors  rx_write_requests
local_ack_timeout_err  req_cqe_error    resp_cqe_error              rnr_nak_retry_err

但是在pod内部同样的路径下是没有 hw_counters 目录下。因为这个目录是在加载驱动时生成，在pod内看不到。

root@pod-dpdk:~/dpdk/x86_64-native-linuxapp-gcc/app# ls /sys/devices/pci0000\:00/0000\:00\:09.0/infiniband/mlx5_0/ports/1/
cap_mask  gid_attrs  gids  lid  lid_mask_count  link_layer  phys_state  pkeys  rate  sm_lid  sm_sl  state

可能的解决办法：
a. 手动 mount /sys/devices/pci0000:00/0000:00:09.0/infiniband/mlx5_0/ports/1/hw_counters/ 到pod内
b. 修改 sriov-network-device-plugin 代码，自动mount上面的目录

测试文件

root# cat dpdk-mlx.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-dpdk
  annotations:
    k8s.v1.cni.cncf.io/networks: host-device1
spec:
  nodeName: node1
  containers:
  - name: appcntr3
    image: l2fwd:v3
    imagePullPolicy: IfNotPresent
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 300000; done;" ]
    securityContext:
      privileged: true
    resources:
      requests:
        memory: 100Mi
        hugepages-2Mi: 500Mi
        cpu: '3'
      limits:
        hugepages-2Mi: 500Mi
        cpu: '3'
        memory: 100Mi
    volumeMounts:
    - mountPath: /mnt/huge
      name: hugepage
      readOnly: False
    - mountPath: /var/run
      name: var
      readOnly: False
  volumes:
  - name: hugepage
    emptyDir:
      medium: HugePages
  - name: var
    hostPath:
      path: /var/run/

参考

https://github.com/Amirzei/mlnx_docker_dpdk/blob/master/Dockerfile
https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/issues/123
https://github.com/DPDK/dpdk/commit/fc40db997323bb0e9b725a6e8d65eae95372446c
https://doc.dpdk.org/guides-18.08/linux_gsg/linux_drivers.html?highlight=bifurcation#bifurcated-driver
https://github.com/kubernetes/kubernetes/issues/60748
https://stackoverflow.com/questions/59290752/how-to-use-device-dev-video0-with-kubernetes

也可参考：k8s mellanox网卡使用dpdk驱动问题总结 - 简书 (jianshu.com)