IB 网卡安全驱动和虚拟化流程

一、安装驱动

1. 安装

tar -zxvf MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu18.04-x86_64.tgz
cd MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu18.04-x86_64
./mlnxofedinstall --force

2. 启动

/etc/init.d/openibd restart
/etc/init.d/opensmd restart
mst start
mst status

二、虚拟化

1. 确认物理机系统是否启动 IOMMU

dmesg | grep -E "DMAR|IOMMU"

2. 修改 grub 启动 amd_iommu=on iommu=pt

vim /etc/default/grub

GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt"

# 更新
update-grub

3. 启动 SR-IOV

mst start

Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices

4. 查看设备状态

mst status

MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4119_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:60:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00

5. 查看 IB 网卡配置文件

mlxconfig -d /dev/mst/mt4119_pciconf0 q


Device #1:
----------

Device type:    ConnectX5
Name:           N/A
Description:    N/A
Device:         /dev/mst/mt4119_pciconf0

Configurations:                              Next Boot
         MEMIC_BAR_SIZE                      0
         MEMIC_SIZE_LIMIT                    _256KB(1)
         HOST_CHAINING_MODE                  DISABLED(0)
         HOST_CHAINING_DESCRIPTORS           Array[0..7]
         HOST_CHAINING_TOTAL_BUFFER_SIZE     Array[0..7]
         FLEX_PARSER_PROFILE_ENABLE          0
         FLEX_IPV4_OVER_VXLAN_PORT           0
         ROCE_NEXT_PROTOCOL                  254
         NON_PREFETCHABLE_PF_BAR             False(0)
         NUM_PF_MSIX_VALID                   True(1)
         NUM_OF_VFS                          16
         SRIOV_EN                            True(1)
         ......

其中
SRIOV_EN ：启动虚拟化
NUM_OF_VFS： 虚拟化网卡的数量

6. 设置 虚拟化网卡数量

mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=24

7. 重启系统，然后从新启动 mst start

8. 修改 /sys/class/infiniband/mlx5_0/device/sriov_numvfs

echo 16 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs

9. 查看当前 IB 网卡 在 lspci 中数量

lspci -D | grep Mellanox

0000:60:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]

10. 查看 IB 网卡当前状态

ibstat

CA 'mlx5_0'
        CA type: MT4119
        Number of ports: 1
        Firmware version: 16.23.1020
        Hardware version: 0
        Node GUID: 0xb88303ffff95a97c
        System image GUID: 0xb88303ffff95a97c
        Port 1:
                State: Down
                Physical state: Disabled
                Rate: 10
                Base lid: 65535
                LMC: 0
                SM lid: 0
                Capability mask: 0x2651e848
                Port GUID: 0xb88303ffff95a97c
                Link layer: InfiniBand

11. 查看 IB 网卡固件中各项参数是否正确

1. VF 数量

cat /sys/class/net/ib0/device/sriov_totalvfs
16

# 如果您没有看到此参数，则 amd_iommu=on 未添加到 grub 文件中。

2. 获取此设备上的当前 VF 数

cat /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs
0

cat /sys/class/net/ib0/device/sriov_numvfs
0

cat /sys/class/net/ib0/device/mlx5_num_vfs
0

12. 再次查看 IB 网卡在 lspci 中数量。可以看到已经虚拟出来的 IB 网卡

lspci -D | grep Mellanox

0000:60:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
0000:60:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.5 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.6 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:00.7 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.5 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.6 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:01.7 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
0000:60:02.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]

13. 查看 IB 网卡使用情况

ibdev2netdev -v

0000:60:00.0 mlx5_0 (MT4119 - IL293304JDV0) HPE InfiniBand EDR 100Gb 1-port 841QSFP28 Adapter fw 16.23.1020 port 1 (DOWN  ) ==> ib0 (Down)
0000:60:00.1 mlx5_1 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:01.2 mlx5_10 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:01.3 mlx5_11 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:01.4 mlx5_12 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:01.5 mlx5_13 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:01.6 mlx5_14 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:01.7 mlx5_15 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:02.0 mlx5_16 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:00.2 mlx5_2 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:00.3 mlx5_3 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:00.4 mlx5_4 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:00.5 mlx5_5 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:00.6 mlx5_6 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:00.7 mlx5_7 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:01.0 mlx5_8 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)
0000:60:01.1 mlx5_9 (MT4120 - NA)  fw 16.23.1020 port 1 (DOWN  ) ==> ib1 (Down)