How to test PCIe SR-IOV in D05
How to test PCIe SR-IOV in D05
-v0.1 2017.2.9 Sherlock init
-v0.2 2017.2.19 Sherlock add pcie_acs_override reference
This doc helps to introduce how to test PCIe SR-IOV in HiSilicon D05 board.
0. preparation
kernel: https://github.com/hisilicon/kernel-dev.git branch: private-topic-sriov-v3-4.10
UEFI: openlab1.0 101 server /home/wangzhou/repo/plinth_uefi/uefi
qemu: https://github.com/eauger/qemu.git branch: v2.7.0-passthrough-rfc-v5
hardware topology:
+---------------+ +----------------------+
| D05 I | | D05 II |
| | | |
| +-----------+ +------------------+ |
| |1P NA PCIe2|<----->|any 10G networking| |
| +-----------+ +------------------+ |
+---------------+ +----------------------+1. compile kernel and UEFI
configure kernel: Add SMMU_V3=y, 82599 PF driver = m, 82599 VF driver = m,
VFIO PCI driver = m (p.s. ACPI boot)
compile kernel image and ko
compile UEFI using: ./uefi-tools/uefi-build.sh -c ./LinaroPkg/platforms.config d05
2. basic test
1. boot up host OS (firstly update UEFI above)[1]
2. copy modules to host OS:
ixgbe.ko ixgbevf.ko mdio.ko vfio_iommu_type1.ko vfio.ko vfio-pci.ko
vfio_virqfd.ko irqbypass.ko
3. prepare host environment:
mkdir /lib/modules/`uname -r`
touch /lib/modules/`uname -r`/modules.order
touch /lib/modules/`uname -r`/modules.builtin
depmod ixgbe.ko ixgbevf.ko mdio.ko vfio_iommu_type1.ko vfio.ko vfio-pci.ko
vfio_virqfd.ko irqbypass.ko
modprobe ixgbe
modprobe ixgbevf
/* trigger one VF, 0002:81:00.0 is the PF in which you want trigger a VF */
echo 1 > /sys/devices/pci0002:80/0002:80:00.0/0002:81:00.0/sriov_numvfs
modprobe -v vfio-pci disable_idle_d3=1
modprobe -r vfio_iommu_type1
modprobe -v vfio_iommu_type1 allow_unsafe_interrupts=1
/* set related PF up */
ifconfig eth26 up
/* 0002:81:10.0 is BDF of VF */
echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
4. run qemu[2]
qemu-system-aarch64 \
-machine virt,gic-version=3 \
-enable-kvm \
-cpu host \
-m 1024 \
-kernel ./Image \
-initrd ./minifs.cpio.gz \
-nographic \
-net none -device vfio-pci,host=0002:81:10.0,id=net0 \
-D trace_log
5. set networking configurations in guest machine and remote machine
run ping and iperf to test
3. more scenarios
1. enable multiple VFs, assigned to one VM
2. enable multiple VFs, assigned to different VMs
3. use VF directly
4. VF and PF communicate
4. performance
1. should at least reach the performance of PF[3]
reference:
[1] should add pcie_acs_override=downstream in kernel command line
[2] how to compile qemu locally in D05:
http://blog.csdn.net/scarecrow_byr/article/details/51494020
[3] should add 82599 patch to test, firstly make sure performance of 82599 PF is
good.