Xen version 3.0 released in 2005 was the first version to support PCI passthrough. You can use PCI passthru to assign a PCI device (NIC, disk controller, HBA, USB controller, firewire controller, soundcard, etc) to a virtual machine guest, giving it full and direct access to the PCI device.
The guest VM (domU) needs to have a driver for the actual PCI device, just like you'd use the PCI device on baremetal (without Xen).
Required software for Xen PCI passthrough:
See below for additional requirements depending which type of Xen PCI passthru you're using (PV or HVM/IOMMU).
Traditional Xen PV PCI passthru to a PV domU gives the guest full control of the PCI device, including DMA access. This can be potentially insecure and unstable, if the guest is malicious or has buggy drivers. The advantage of this PV PCI passthru method is that it has been available for years in Xen, and it doesn't require any special hardware or chipset (it doesn't require hardware IOMMU (VT-d) support from the hardware).
Additional requirements for Xen PCI passthru to PV domU:
PV domU kernel needs to have the Xen PCI frontend driver loaded for PCI passthru to work! This driver is called xen-pcifront in pvops kernels.
You need to add additional kernel options to the domU kernel cmdline to enable Xen PCI passthru, see below.
You can also use the more safe and secure hardware IOMMU (VT-d) PCI passthru for PV guests, see below for that.
For DMA access to work for pci devices in a PV guest it is required to enable the swiotlb in the domU (guest VM) kernel. (for some explanation about the swiotlb see http://lwn.net/Articles/91870/)
Add this option to your domU (guest VM) kernel boot options (cmdline arguments) list :
swiotlb=force
NOTE: This option is only required for older PV guest kernel versions (linux-2.6.18-xen, RHEL5, SLES10/11, Debian etch/lenny).
upstream kernel.org (pvops) Linux domU kernels requires just this option:
iommu=soft
Requirements for Xen PCI passthru to HVM guest:
Hardware IOMMU (Intel VT-d or AMD IOMMU) is required from the CPU/motherboard/chipset/BIOS.
Note that IOMMU/VT-d is different additional feature than the normal CPU virtualization extensions (VMX/SVM) required for running Xen HVM guests!! Intel VT-x or AMD-V provide the VMX or SVM CPU flags (CPU virtualization extensions), but these flags don't give you IOMMU/VT-d support! IOMMU support is not yet available in many chipsets (as of the beginning of the year 2010).
To verify you have IOMMU support enabled:
Please see the VTdHowTo wiki page for more information about PCI passthru to Xen HVM guest.
Xen HVM guest doesn't need to have a special kernel or special pcifront drivers in it for the PCI passthru to work.
module /boot/vmlinuz-2.6.32.10 root=/dev/sda1 ro nomodeset xen-pciback.hide=(08:05.0)
module /boot/vmlinuz-2.6.32.10 root=/dev/sda1 ro nomodeset xen-pciback.hide=(01:00.0)(00:02.0)
Note important: Make sure the device shows up in the "xm pci-list-assignable-devices" list! Don't continue before you've gotten that properly working.
Edit "/etc/xen/<guest>" cfgfile and add the following line to enable PCI passthru
pci = [ '01:00.0' ]
pci = [ '01:00.0', '00:02.0' ]
You can verify the PCI passthru status with: "xm pci-list <guest>"
You can also do PCI passthru online to already running Xen guest. The PCI device must be ready for passthru before doing the hotplug, ie. it has to show up in the " xm pci-list-assignable-devices" list. This hotplug method works for both PV and HVM guests. Commands for Xen guest PCI hotplug/hot-unplug are:
xm pci-attach <guest> <pci device> <guest virtual slot number>
xm pci-detach <guest> <pci device> <guest virtual slot number>
List of Xen pciback modes that you can set in the kernel configuration (.config file) in xen/stable-2.6.32.x kernel:
Note that in upstream Linux 3.1.0 and later versions you can set PASS/VCPI as a module/driver option when loading the driver!
You can use the following on dom0 Linux kernel command line in grub.conf (if xen-pciback is built-in to the kernel):
xen-pciback.passthrough=1
or the following if loading xen-pciback driver as a module:
modprobe xen-pciback passthrough=1
In Linux 3.1+ this will give the same behavious as earlier CONFIG_XEN_PCIDEV_BACKEND_PASS .config option.
When the guest has PCI passthru devices in use, operations like save/restore/migration are not possible. You have to detach (unplug) the passthru device before save, restore or live migration is possible.
Please see the XenVGAPassthrough wiki page for more information about VGA graphics card passthru.
Yes, you can use the more safe and secure hardware IOMMU (VT-d) passthru also for PV guests, instead of the normal Xen PV PCI passthru. As a default Xen uses the normal (non-IOMMU) PV passthru for PV guests.
You need to add "iommu=pv" boot option for Xen hypervisor (xen.gz) in grub.conf and reboot. After rebooting verify from "xm dmesg" that IO virtualization for PV guests gets enabled. Naturally you must have hardware IOMMU (Intel VT-d or AMD-V) support for this to work.
From Xen 4.0.1 and above, the iommu=pv is not necessary, as it is turned on by default as long as vt-d is presented during boot.
pvops dom0 kernels renamed the xen related driver-modules in Dec 2009 to have xen- prefix in them, so pciback driver is now called xen-pciback. So please use "xen-pciback.hide" option instead of just "pciback.hide".
Yes, it's possible with pvops dom0 kernel xen pci backend sysfs interface. Please see this email for instructions: http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00448.html
If using linux-2.6.18-xen, add these options to grub.conf for the 2.6.18.8 dom0 kernel which should fix the alignment:
guestdev=01:00.0,01:02.0 reassign_resources
replace "01:00.0" and "01:02.0" with your actual PCI devices you want to passthru. Note the "," to separate the entries.
There was a change in Apr 2009 in linux-2.6.18-xen (http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/a3ad7a5f2dcd) that changed the syntax.. the earlier/old syntax for linux-2.6.18-xen is:
pciback.permissive pciback.hide=(01:00.0)(02:01.0) reassigndev
If you're using Linux 2.6.31 or newer pvops dom0 kernel then there's no guestdev/reassign_resources, but instead you use:
xen-pciback.permissive xen-pciback.hide=(08:05.0)(09:06.1) pci=resource_alignment=08:05.0;09:06.1
If you're using Linux 2.6.31 or newer dom0 kernel based on the Novell/SLES/OpenSuse Xenlinux forward-ported patches, then you use this syntax:
pciback.permissive pciback.hide=(00:1d.7)(00:1a.0)(00:1a.1)(00:1a.7)(00:1b.0) pci=resource_alignment=00:1a.7;00:1d.7
Note the ";" to separate multiple PCI ID entries for "pci=resource_alignment".
If using GRUB2, and using resource_alignment for multiple devices, you need to wrap the resource_alignment with single quotes like this:
'pci=resource_alignment=00:1a.7;00:1d.7'
Otherwise GRUB2 will parse the line wrong and you won't get any resource_alignment! For more info see: http://lists.xensource.com/archives/html/xen-users/2011-09/msg00360.html .
This error usually happens when you're trying to passthru only a single function from a multi-function device (for example a dual-port nic), or only one of the devices behind the same PCI bridge. This is not allowed by the Intel VT-d specification. Please see this email for the explanation of this issue: http://lists.xensource.com/archives/html/xen-devel/2010-01/msg00870.html and the patch implementing these FLR methods: http://xenbits.xensource.com/xen-unstable.hg?rev/e61978c24d84
If you want to manually override this in Xen 4.0.0 or newer you can specify "pci-passthrough-strict-check no" in /etc/xen/xend-config.sxp, and after restarting xend passthru code won't give this error anymore. In some (many?) cases PCI passthru can work after this change.
If the PCI device is a single-function device, you can also move it to a different PCI slot to workaround the issue.
With Xen 3.4.x and 3.3.x versions you can apply a "disable FLR" patch to workaround this issue: http://lists.xensource.com/archives/html/xen-devel/2008-10/binAofZNDKlrU.bin and discussion about it here http://lists.xensource.com/archives/html/xen-devel/2008-10/msg00280.html
Add "iommu=verbose" option for Xen hypervisor (xen.gz) in grub.conf and reboot. After rebooting read "xm dmesg" log (or set up a serial console). As a default Xen 4.0.0 is not verbose about IOMMU initialization and related ACPI DMAR table parsing.
Yes. Starting with kernel.org Linux 2.6.37 Xen PV domU PCI passthrough is supported out-of-the-box, ie. xen-pcifront driver is included in the standard kernel!
There are also multiple additional git trees and branches available with the pvops Xen PCI Frontend driver included.
Jeremy's xen.git in git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git has Xen pcifront driver at least in the following branches:
Konrad's xen.git in git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git has Xen pcifront driver in the following branch (note these branches are only suitable for domU use):
You can get the branches like this:
$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git linux-domU $ cd linux-domU $ git checkout -b temp origin/pv/merge.2.6.34
Bugs:
[Fixed with Xen c/s 23249 "pv-grub: Fix for incorrect dom->p2m_host[] list initialization"] Starting the DomU using pvgrub with 'iommu=soft swiotlb=force' breaks pvgrub.
See http://lists.xensource.com/archives/html/xen-devel/2010-04/msg00285.html for Apr 2010 update of pcifront/swiotlb patches.
See http://lists.xensource.com/archives/html/xen-devel/2010-04/msg01065.html for Apr 2010 updated pcifront/pciback patches with MSI/MSI-X support.
Here's a list of the kernels that have the Xen pcifront driver included. This list might not be complete.
Novell/OpenSuse forward-ported Xenlinux patches, rebased by Andrew Lyon for 2.6.29, 2.6.31, 2.6.32, 2.6.33, etc
See XenKernelFeatures wiki page for more information about supported features in Xen kernels.
Unfortunately many motherboards ship with broken BIOSes (for example incorrect ACPI DMAR, DRHD or RMRR tables) that causes Xen to disable IO virtualization as a security measure, or to prevent crashes from happening later on.
You can check if Xen enabled IO virtualization by running "xm dmesg" command and reading through the log. There's a line about IO virtualization telling if it's enabled or disabled. You need to have at least Xen 3.4 or newer for IOMMU (VT-d) to work.
If IO virtualization gets disabled, but it's available on your hardware, you should try these steps to troubleshoot it:
Intel developers also want to know about broken IOMMU/VT-d BIOS implementations, see this email: http://lists.xensource.com/archives/html/xen-devel/2010-01/msg00841.html, so let them know all the details about your hardware and software if you have broken BIOS.
Please see BDFNotation wiki page for more information.
Well.. yes. This is totally unsupported, and it only works for the first started guest. Oh, and you're totally on your own with this patch: http://lists.xensource.com/archives/html/xen-devel/2009-09/msg00018.html . Good luck
Run "lspci -vv" (in dom0) and check if the device has "FLReset+" in the DevCap field.
pci-stub can be used only with Xen HVM guest PCI passthru, so it's recommended to use pciback instead, which works for both PV and HVM guests.
When passing PCI devices rather then PCIe device, it is necessary to include all the sub devices before PCI passthrough works. E.g.
lspci
06:03.0 FireWire (IEEE 1394): Agere Systems FW322/323 (rev 70)
For user who wish to have 06:00.0-2 to pass to domU, it is necessary to add following line to the /boot/grub/grub.cfg in dom0 kernel xen-pciback.hide=(06:00.0)(06:00.1)(06:00.2)(06:00.3)(06:03.0)
When (06:03.0) is left out, the pci passthrough won't work!!
In the /etc/xen/abc.cfg file, the following line is fine pci = ['06:00.0', '06:00.1', '06:00.2']
The above example is used for Xen 4.0.1 with Debian Squeeze as dom0.
#!/bin/bash
if [ $# -eq 0 ]; then
echo "Require PCI devices in format: <domain>:<bus>:<slot>.<function>"
echo "Eg: $(basename $0) 0000:00:1b.0"
exit 1
fi
modprobe pciback
for pcidev in $@; do
if [ -h /sys/bus/pci/devices/"$pcidev"/driver ]; then
echo "Unbinding $pcidev from" $(basename $(readlink /sys/bus/pci/devices/"$pcidev"/driver))
echo -n "$pcidev" > /sys/bus/pci/devices/"$pcidev"/driver/unbind
fi
echo "Binding $pcidev to pciback"
echo -n "$pcidev" > /sys/bus/pci/drivers/pciback/new_slot
echo -n "$pcidev" > /sys/bus/pci/drivers/pciback/bind
done