Jupiter在成都

vpp的网卡直通配置和SRIOV配置方法

本文总结了Centos-KVM作为宿主机，centos-vpp作为客户机时的网卡passthrough和SRIOV的配置方法

配置网卡直通

一、宿主机上修改GRUB，增加intel_iommu=on。刷新GRUB。然后重启宿主机。

[root@kvm-02 ~]# vi /etc/default/grub

[root@kvm-02 ~]# grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

Generating grub configuration file ...

Found linux image: /boot/vmlinuz-3.10.0-693.el7.x86_64

Found initrd image: /boot/initramfs-3.10.0-693.el7.x86_64.img

Found linux image: /boot/vmlinuz-0-rescue-20fe28cd4f4b4fa1b140c6a72d10ae05

Found initrd image: /boot/initramfs-0-rescue-20fe28cd4f4b4fa1b140c6a72d10ae05.im

done

（注：刷新grub的方法与系统引导方式有关，如果是legacy模式，就是/boot/grub2/grub.cfg，如果是uefi模式，则为/boot/efi/EFI/centos/grub.cfg）

[root@kvm-02 ~]# reboot

等启动完了之后

[root@kvm-02 ~]# cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.10.0-693.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap intel_iommu=on isolcpus=20-23 nohz_full=20-23 rcu_nocbs=20-23 nmi_watchdog=0 selinux=0 intel_pstate=disable nosoftlockup rhgb quiet

[root@kvm-02 ~]#

二、通过virsh命令将PCI从宿主机分离

[root@kvm-02 ~]# lspci -nn | grep net

09:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ [8086:37d3] (rev 09)

09:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ [8086:37d3] (rev 09)

09:00.2 Ethernet controller [0200]: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ [8086:37d3] (rev 09)

09:00.3 Ethernet controller [0200]: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ [8086:37d3] (rev 09)

2f:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

2f:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

31:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)

31:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 [8086:158b] (rev 02)

58:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

58:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

86:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]

86:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]

af:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

af:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)

virsh nodedev认的PCI格式稍微有点区别，用virsh再显示一遍

[root@kvm-02 ~]# virsh nodedev-list --tree | grep 09

| +- pci_0000_09_00_0

| +- pci_0000_09_00_1

| +- pci_0000_09_00_2

| +- pci_0000_09_00_3

+- pci_0000_05_09_0

+- pci_0000_05_09_1

+- pci_0000_05_09_2

+- pci_0000_05_09_3

+- pci_0000_05_09_4

+- pci_0000_05_09_5

+- pci_0000_05_09_6

+- pci_0000_05_09_7

+- pci_0000_2e_09_0

| | +- block_sdc_MTFDDAK480TBY_1AR1ZA_01PE061D7A09450LEN_1CC00A37

| | +- block_sdd_MTFDDAK480TBY_1AR1ZA_01PE061D7A09450LEN_1CFD6740

+- pci_0000_85_09_0

+- pci_0000_85_09_1

+- pci_0000_85_09_2

+- pci_0000_85_09_3

+- pci_0000_85_09_4

+- pci_0000_85_09_5

+- pci_0000_85_09_6

+- pci_0000_85_09_7

+- pci_0000_ae_09_0

将设备从宿主机分离出来

[root@kvm-02 ~]# virsh nodedev-dettach pci_0000_09_00_0

已分离设备 pci_0000_09_00_0

[root@kvm-02 ~]# virsh nodedev-dettach pci_0000_09_00_1

已分离设备 pci_0000_09_00_1

三、通过virsh edit修改虚拟机的xml配置，往虚拟机挂载PCI; 修改完虚拟机配置文件后,运行虚拟机.

比如virsh edit vm115_vnf，在devices里新增hostdev配置（注意bus slot function的编号是和pci编号对应着来的）

bus='0x09' slot='0x00' function='0x0' />

修改完后保存退出，再 virsh start vm115_vnf

四、虚拟机里lspci可以看到新挂载的PCI，给新的PCI增加驱动后，vpp即可显示出相应的接口

如下：00:0d.0和00:12.0是新挂上的PCI

[root@vnf1-0 ~]# lspci

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)

00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]

00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]

00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)

00:02.0 VGA compatible controller: Cirrus Logic GD 5446

00:03.0 Ethernet controller: Red Hat, Inc Virtio network device

00:04.0 Ethernet controller: Red Hat, Inc Virtio network device

00:05.0 Ethernet controller: Red Hat, Inc Virtio network device

00:06.0 Ethernet controller: Red Hat, Inc Virtio network device

00:07.0 Ethernet controller: Red Hat, Inc Virtio network device

00:08.0 Ethernet controller: Red Hat, Inc Virtio network device

00:09.0 Ethernet controller: Red Hat, Inc Virtio network device

00:0a.0 Ethernet controller: Red Hat, Inc Virtio network device

00:0b.0 Ethernet controller: Red Hat, Inc Virtio network device

00:0c.0 Ethernet controller: Red Hat, Inc Virtio network device

00:0d.0 Ethernet controller: Intel Corporation Device 37d3 (rev 09)

00:0e.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)

00:0e.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)

00:0e.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)

00:0e.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)

00:0f.0 Communication controller: Red Hat, Inc Virtio console

00:10.0 SCSI storage controller: Red Hat, Inc Virtio block device

00:11.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon

00:12.0 Ethernet controller: Intel Corporation Device 37d3 (rev 09)

给新挂的PCI增加igb_uio驱动后，dpdk可以识别出接口。从而vpp就能看到接口了

modprobe uio

insmod /home/dpdk-stable-18.02.2/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko

/home/dpdk-stable-18.02.2/usertools/dpdk-devbind.py --bind=igb_uio 0000:00:12.0 0000:00:0d.0 0000:00:0c.0 0000:00:0b.0 0000:00:0a.0 0000:00:09.0 0000:00:08.0 0000:00:07.0 0000:00:06.0 0000:00:05.0 0000:00:04.0

五、virsh nodedev-reattach pci_0000_09_00_0命令可以重新加载被分离的PCI

[root@kvm-02 net]# virsh nodedev-reattach pci_0000_09_00_0

重新附加设备 pci_0000_09_00_0

[root@kvm-02 net]# virsh nodedev-reattach pci_0000_09_00_1

重新附加设备 pci_0000_09_00_1

配置SRIOV

1、linux的引导参数里使能intel_iommu

intel_iommu=on

2、生成VF

linux内核3.8以下的和3.8以上的有区别。3.8以下版本先卸载驱动模块，再重新拉起驱动模块并附加max_vfs参数，网上较多这种配置方法指导。3.8以上的如下：

先查看内核版本

[root@kvm-02 rc.d]# cat /proc/version

Linux version 3.10.0-862.11.6.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Tue Aug 14 21:49:04 UTC 2018

[root@kvm-02 rc.d]#

通过ifconfig查看想生成VF的网卡名。

或者直接查看device文件

[root@kvm-02 ~]# cd /sys/class/net

[root@kvm-02 net]# ls

br0 br1 br10 br11 br12 br2 br3 br4 br5 br6 br7 br8 br9 eno1 eno2 eno3 eno4 enp0s20f0u1u6 enp134s0f0 enp134s0f1 enp175s0f0 enp175s0f1 enp47s0f0 enp47s0f1 enp49s0f0 enp49s0f1 enp88s0f0 enp88s0f1 lo virbr0 virbr0-nic

我需要使用的是网卡eno1和eno2，（4代表要生成4个VF，device最大支持多少VF可通过命令查看： cat /sys/class/net/device name/device/sriov_totalvfs）

echo 4 > /sys/class/net/eno1/device/sriov_numvfs

echo 4 > /sys/class/net/eno2/device/sriov_numvfs

查看pci，可以看到生成的VF（virtual function）

[root@kvm-02 rc.d]# lspci | grep Eth

09:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 09)

09:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 09)

09:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 09)

09:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 09)

09:02.0 Ethernet controller: Intel Corporation X722 Virtual Function (rev 09)

09:02.1 Ethernet controller: Intel Corporation X722 Virtual Function (rev 09)

09:02.2 Ethernet controller: Intel Corporation X722 Virtual Function (rev 09)

09:02.3 Ethernet controller: Intel Corporation X722 Virtual Function (rev 09)

09:06.0 Ethernet controller: Intel Corporation X722 Virtual Function (rev 09)

09:06.1 Ethernet controller: Intel Corporation X722 Virtual Function (rev 09)

09:06.2 Ethernet controller: Intel Corporation X722 Virtual Function (rev 09)

09:06.3 Ethernet controller: Intel Corporation X722 Virtual Function (rev 09)

2f:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

2f:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

31:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)

31:00.1 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)

58:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

58:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

86:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe

86:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe

af:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

af:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

[root@kvm-02 rc.d]#

三、HOST上给VF配置mac地址

[root@kvm-02 /]# ip link set eno1 vf 0 mac 00:A0:00:00:01:00

[root@kvm-02 /]# ip link set eno2 vf 0 mac 00:A0:00:00:02:00

[root@kvm-02 /]# ip link show eno1

3: eno1: mtu 1500 qdisc mq master br1 portid 7cd30a5bdb98 state DOWN mode DEFAULT qlen 1000

link/ether 7c:d3:0a:5b:db:98 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 00:a0:00:00:01:00, spoof checking on, link-state auto, trust off

vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

[root@kvm-02 /]# ip link show eno2

5: eno2: mtu 1500 qdisc mq master br2 portid 7cd30a5bdb99 state DOWN mode DEFAULT qlen 1000

link/ether 7c:d3:0a:5b:db:99 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 00:a0:00:00:02:00, spoof checking on, link-state auto, trust off

vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

四、将生成的VF从HOST分离出来

先查看有哪些PCI

[root@kvm-02 rc.d]# virsh nodedev-list --tree | grep 09

| +- pci_0000_09_00_0

| +- pci_0000_09_00_1

| +- pci_0000_09_00_2

| +- pci_0000_09_00_3

| +- pci_0000_09_02_0

| +- pci_0000_09_02_1

| +- pci_0000_09_02_2

| +- pci_0000_09_02_3

| +- pci_0000_09_06_0

| +- pci_0000_09_06_1

| +- pci_0000_09_06_2

| +- pci_0000_09_06_3